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Abstract 

The  magnitude  of  money  involved  in  the  acquisition  of  defense  systems  and  the 
public  scrutiny  resulting  from  cost  overruns  and  program  failures  make  cost  management 
con^tence  critical  to  program  success.  This  research  examined  the  cost  management 
competencies  required  of  defense  program  managers.  A  cost  management  competency 
model  was  developed  from  a  foundation  of  past  research.  The  model  was  evaluated 
through  a  mail  survey  of  682  intermediate  and  senior  level  military  program  managers  in 
Air  Force  Materiel  Cotmnand.  The  results  provided  by  the  330  respondents  indicate  that 
29  of  the  47  competencies  in  the  model  were  valuable  to  the  program  managers.  The 
results  indicate  that  both  intermediate  and  senior  level  program  managers  rely  more  on 
understanding  cost  management  concepts  than  on  being  able  to  complete  the  tasks 
themselves.  Data  analysis  identified  differences  in  the  perceived  importance  and  frequency 
of  use  of  some  competencies  based  on  management  education,  organization  type,  primary 
program  activity,  and  position  certification  level.  The  results  have  direct  implications  for 
the  development  of  defense  program  managers.  Education  and  training  programs  can  be 
improved  by  focusing  on  the  cost  management  competencies  that  will  be  most  valuable  to 
program  managers  in  the  field. 
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COST  MANAGEMENT  COMPETENCIES 
FOR  DEPARTMENT  OF  DEFENSE  PROGRAM  MANAGERS 

L  Introduction 

Background 

Over  35  percent  ($92  billion  in  1993)  of  the  Department  of  Defense  (DoD)  budget 
has  historically  been  allocated  to  acquisition  organizations  within  the  DoD  for  the 
purposes  of  accomplishing  the  research,  development,  and  procurement  of  systems  in 
support  of  national  defense  (Cheney,  1993: 143).  However,  the  public’s  perception  of 
how  these  funds  are  managed  has  been  tainted  by  numerous  weapon  system  acquisition 
“scandals”  which  resulted  in  overruns  costing  taxpayers  billions  of  dollars. 

"Creating  a  Professional  Acquisition  Work  Force"  by  Congressman  Nicholas 
Mavroules  gives  insight  into  the  history  of  problems  associated  with  the  acquisition  of 
defense  systems.  In  1991,  Representative  Mavroules  was  the  chairman  of  the 
Subcommittee  on  Investigations,  Committee  on  Armed  Services,  U.S.  House  of 
Representatives.  In  his  article.  Representative  Mavroules  states  that  the  scandals  that 
arose  within  the  acquisition  community  during  the  1980s  were  nothing  new.  They  date 
back  to  the  Navy's  contract  for  its  first  warship,  the  USS  Constitution,  which  overran  by 
175  percent  (Mavroules,  1991:15).  Three  more  recent  examples  of  DoD  acquisition 
overruns  and  cost  mismanagement  are  described  in  the  following  paragraphs. 
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The  National  Aerospace  Plane  (NASP)  was  conceived  as  a  cooperative  effort 
between  the  Air  Force  ,  the  Navy,  and  the  National  Aeronautics  and  Space  Administration 
(NASA)  to  design  a  futuristic,  hypersonic,  high-altitude  air  vehicle.  Politicians  have 
criticized  the  NASP  program  because  of  the  inaccuracy  of  the  cost  estimates.  For  NASP, 
“total  program  cost  now  is  estimated  at  $12-15  billion,  roughly  three  times  the  initial 
estimates”  (Scott,  1993:23).  Future  funding  is  expected  to  be  more  consistent  with  the 
original  estimates,  which  fall  well  short  of  updated  estimates.  Thus,  the  future  of  the 
NASP  program  is  uncertain  (Oliver,  1994). 

The  Navy’s  A- 12  was  intended  to  be  the  stealth  attack  aircraft  that  would  take  the 
Navy  into  the  next  century.  The  contract  for  the  A- 12  was  terminated  for  default  in  1991 
(USGAO,  1991:2).  Significant  contributors  to  that  program’s  downfall  included  the 
inapprq)riate  use  of  a  fixed  price  contract  for  development  and  the  uncertainty  created  by 
the  improper  use  of  cost  and  schedule  control  data  for  reporting  program  estimates  at 
completion  (Morrison,  1991:31;  Christensen  and  Heise,  1993).  As  of  July  1991,  the  Navy 
A- 12  contractor  team  had  filed  a  termination  for  convenience  proposal  and  claim  for  a 
settlement  worth  between  $1.3  billion  and  $1.9  billion.  The  contractors  are  seeking  all 
incurred  costs,  a  reasonable  profit,  and  settlement  expenses  (USGAO,  1992:6). 

The  C-17  is  the  Air  Force’s  next  generation  transpon  aircraft  Technical  shortfalls 
in  the  C- 17  resulted  in  excess  of  $1  billion  in  cost  overruns.  In  addition,  certain 
government  program  personnel,  including  the  former  program  manager,  were  disciplined 
for  advancing  the  prime  contractor  nearly  $500  million  in  unjustified  progress  payments 
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(Morrocco,  1993b:51).  As  of  January  1993,  McDonnell  Douglas  was  preparing  over  $1 
billion  in  claims  for  costs  incurred  on  the  C-17  program  (Morrocco,  1993b:51). 


Table  1  represents  a  sample  of  pertinent  articles  and  General  Accounting  Office 
reptnrts  identifying  various  cost  management  deficiencies  as  a  significant  source  of  DoD 
acquisition  failures  such  as  those  mentioned  above. 


Table  1.  DoD  Cost  Management  Competency  Deficiencies 
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Table  1  illustrates  the  wide  variety  of  cost  management  competencies  that,  having 
been  neglected,  resulted  in  cost  overruns  and  program  failures.  In  particular,  contractor 
financing,  cost  estimating,  scope  changes,  corrective  actions,  and  operating  within  the 
DoD  political  environment  have  been  frequently  cited  as  areas  of  competency  deficiencies. 

Media  reports  of  cost  mismanagement  on  the  NASP,  C-17,  A- 12  and  other 
programs  prompted  investigations  into  every  aspect  of  the  business  of  defense  systems 
acquisition.  Since  World  War  II,  there  have  been  at  least  six  major  commissions  that  have 
looked  into  the  problems  of  military  acquisition;  the  Hoover  Commissions  of  1949  and 
1955,  the  Fitzhugh  Commission  of  1970,  the  Commissions  on  Government  Procurement 
in  1972,  the  Grace  Commission  of  1983,  and  the  Packard  Commission  of  1986.  In  1986, 
the  Packard  Commission,  also  known  as  the  President's  Blue  Ribbon  Commission  on 
Defense  Management,  issued  a  final  report  stating  that  the  acquisition  workforce  was 
undertrained,  underpaid,  and  inexperienced  (Cheney,  1989:12;  President,  1986).  The 
other  commissions  also  recognized  the  need  for  a  competent  work  force,  yet  their  findings 
were  never  implemented  in  the  form  of  legislative  and  policy  improvements  (Mavroules, 
1991:18).  Until  1991,  the  legislation  focused  primarily  on  only  two  of  three  elements 
within  the  acquisition  system:  the  process  and  the  structure.  In  1991,  Congress  acted  on 
the  third  element  -  the  people  (Mavroules,  1991:16). 

Following  numerous  inquiries  and  investigations,  legislators  agreed  with  the  Packard 
Commission's  finding  that  the  lack  of  experience  and  appropriate  training  of  acquisition 
personnel  was  a  significant  shortfall  within  the  acquisition  system.  In  an  effort  to  reduce 
costly  mistakes  in  the  arena  of  defense  systems  acquisition.  Congress  passed  the  Defense 
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Acquisition  Workforce  Improvement  Act  of  1990  (DAWIA).  This  legislation  required 
DoD  to  establish  a  professional  acquisition  workforce  with  training  and  experience 
commensurate  with  the  responsibilides  of  the  many  types  of  acquisition  positions. 

DAWIA  also  required  the  establishment  of  a  highly  trained,  educated,  and  experienced 
acquisition  corps  to  fill  the  most  critical  acquisition  positions  (US  Congress,  1990:Sec. 
1722, 23,  31). 

Department  of  Defense  Directive  (DODD)  5(X)0.52,  Defense  Acquisition 
Education,  Training,  and  Career  Development  Program,  and  DOD  5000.52M,  Career 
Development  Program  For  Acquisition  Personnel,  implemented  DAWIA.  These 
documents  required  the  armed  services  to  establish  and  maintain  systems  by  which 
acqitisition  personnel  would  be  certified  at  various  levels,  based  on  their  acquisition 
experience,  education,  and  training  (DODD  5000.52, 1991:2).  The  Air  Force  certification 
program  is  currently  called  the  Acquisition  Professional  Development  Program  (APDP) 
(IG,  1992).  This  program  replaced  the  previous  Air  Force  initiative  known  as  the 
Acquisition  Management  Professional  Development  Program  (AMPDP).  Table  2 
highlights  the  differences  between  the  AMPDP  requirements  for  program  managers  and 
those  subsequently  required  by  DOD  5000.52M. 

The  Air  Force  was  pleased  with  the  AMPDP  certification  requirements  because 
the  AMPDP  certification  program  “successfully  identified  personnel  with  the  qualifications 
needed  to  manage  complex  acquisitions”  (IG,  1992:10).  However,  as  Table  2  indicates, 
DOD  5000.52M  certification  requirements  were  not  as  stringent. 
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Many  Air  Force  acquisition  personnel  felt  the  resulting  uniform,  but  rather 
easily  met,  OSD  certification  requirements  would  not  stimulate  development  of 
the  professional  acquisition  corps  intended  by  the  DAWIA  and  envisioned  by 
senior  Air  Force  Leadership.  (IG,  1992:9) 


Table  2.  Program  Management  Professional  Development 
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The  Air  Force  sought  to  compensate  for  the  new,  lower  standards  imposed  by 
DOD  5000.52M  by  requiring  higher  standards  in  their  APDP,  the  replacement  for  the 
AMPDP.  Specifically,  the  original  APDP  requirements  included  four  years  acquisition 
experience  for  Level  n  certification  and  eight  years  acquisition  experience  for  Level  in 
certification.  DOD  5000. 52M  only  required  two  years  acquisition  experience  for  Level  II 
and  four  years  for  Level  HI.  The  APDP  also  required  completion  of  the  Defense  Systems 
Management  College  Program  Management  Course  (DSMC  PMC).  This  course  was  not 
required  by  DOD  5000.52M  (IG,  1992:12).  The  Air  Force’s  attempts  to  use  these  higher 
standards  in  their  APDP  than  the  other  services  were  using  in  their  certification  programs 
met  strong  resistance  from  the  DoD  because  the  different  standards  made  it  more  difficult 
to  assess  the  qualifications  of  the  services’  acquisition  workforces,  to  compare  individuals 
among  the  services,  and  to  measure  compliance  with  DAWIA. 

The  Air  Force  fought  for  higher  standards  because  Air  Force  acquisition  personnel 
were  typically  more  experienced  and  had  more  acquisition  education  and  training  than 
their  contemporaries  in  the  Army  and  Navy,  and  the  Air  Force  believed  the  higher 
standards  would  provide  a  better  measure  of  an  individual’s  ability  to  manage  complex 
acquisitions  (IG,  1992:9).  However,  the  Air  Force’s  attempts  to  maintain  standards  above 
those  required  under  DOD  5()00.52M  were  abandoned  in  early  1994  (Druyun,  1993). 
Arguments  for  uniformity  among  services  prevailed  over  the  Air  Force’s  desire  for  higher 
standards.  Air  Force  APDP  requirements  are  now  fully  consistent  with  the  DoD 
requirements  and  those  of  the  other  services. 
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The  APDP  certification  criteria  are  general  in  nature.  There  are  no  explicit 
requirements  for  training,  education,  or  experience  for  program  managers  in  specific  areas 
like  cost  management.  Therefore,  the  tools  to  manage  education  and  training  programs  at 
a  detailed  level  are  limited.  Competency  models  in  specific  areas  like  cost  management 
would  provide  the  needed  tools  for  creating  and  managing  programs  to  develop 
acquisition  workforce  competence.  The  American  Heritage  Dictionary  (Heritage, 
1982:301)  defines  the  word  competent  as: 

1.  Properly  or  well  qualitied;  capable. 

2.  Adequate  for  the  purpose;  sufficient. 

3.  Legally  qualified  or  fit;  admissible. 

For  the  purposes  of  this  thesis,  the  term  competency  will  be  interpreted  as  a  capability 
required  for  proficient  program  management  When  program  managers  possess  such  an 
ability,  that  means  they  possess  the  skills  and  knowledge,  as  well  as  the  physical  and 
mental  characteristics  necessary  to  successfully  complete  a  task  or  understand  a  particular 
aspect  of  cost  management  (Note:  The  terms  program  manager  and  project  manager  are 
used  interchangeably  throughout  this  thesis,  as  are  program  management  and  project 
management) 

Accurate  identification  of  what  program  managers  need  to  know  and  do  can 
maximize  the  benefit  of  time  and  money  spent  on  education  and  training  programs.  For 
many  program  managers,  formal  cost  management  training  may  be  limited  to  the  few 
hours  provided  during  general  acquisition  courses  like  PMT  101,  Fundamentals  of 
Systems  Acquisition  Management,  and  PMT  201,  Intermediate  Systems  Acquisition 
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(DAU,  1993:74).  However,  for  an  aspect  of  program  management  as  critical  as  cost 
management^  these  overviews  may  not  be  sufficient. 

Although  the  criteria  of  experience,  education,  and  training  are  necessary  to  reflect 
an  employee’s  exposure  to  various  aspects  of  the  acquisition  process,  they  may  not  be 
sufficient  to  distinguish  the  qualifications  of  one  acquisition  professional  from  another,  and 
they  may  not  accurately  reflect  the  demands  placed  on  program  managers  in  their  day-to- 
day  activities.  These  criteria  simply  measure  attendance  at  professional  development 
courses  and  job  locations;  they  do  not  measure  a  person’s  competence  in  acquisition 
activities.  Incorporation  of  competencies  into  the  education,  training,  and  certification 
systems  would  pull  employees  to  the  level  of  competence  needed  to  successfully  manage 
complex  acquisitions.  In  other  words,  the  standards  governing  service  needs  should  not 
be  driven  by  what  the  services  cunently  have  or  what  the  training/education  system  is 
currently  able  to  provide.  The  standards  need  to  originate  from  the  demands  that  are 
placed  on  acquisition  personnel  in  their  day  to  day  activities. 

This  thesis  is  based  on  the  premise  that,  although  education,  training,  and 
experience  are  not  sufficient  to  guarantee  competence,  they  are  common  vehicles  through 
which  individuals  develop  competence.  Although  quantifying  a  person’s  competence  is 
difficult  to  do,  identifying  those  competencies  required  for  success  is  achievable  and 
necessary  for  the  purpose  of  developing  training  programs,  education,  career  progression 
plans,  and  useful  certiflcation  criteria. 
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The  1989  Defense  Management  Report  to  the  President  clearly  states  the  general  problem 
area  addressed  by  this  thesis: 


The  defense  acquisition  workforce  mingles  civilian  and  military  expertise  in 
numerous  disciplines  for  management  and  staffing  of  the  world’s  largest 
procurement  organization.  Each  year  billions  of  dollars  are  spent  more  or  less 
efficiently,  based  on  the  competence  and  experience  of  these  personnel.  Yet, 
compared  to  its  industry  counterparts,  this  workforce  is  undertrained,  underpaid, 
and  inexperienced.  Whatever  other  changes  may  be  made,  it  is  vitally  important 
to  enhance  the  quality  of  the  defense  acquisition  workforce  —  both  by  attracting 
qualified  new  personnel  and  by  improving  the  training  and  motivation  of  current 
personnel.  (Cheney,  1989:12) 

Models  identifying  the  competencies  required  of  program  managers  could  provide 
valuable  information  for  establishing  certification  criteria  and  for  designing  training  and 
education  programs  for  program  managers.  This  research  effon  evaluated  competencies 
for  the  cost  management  area.  For  the  puiposes  of  this  research,  intermediate  and  senior 
level  program  managers  are  defined  as  those  program  managers  who  occupy  positions 
coded  as  APDP  Program  Management  Level  II  and  Level  IE,  respectively.  Using  this 
definition,  a  Level  II  program  manager  must  have  at  least  two  years  acquisition 
experience,  and  a  Level  HI  program  manager  must  have  at  least  four  years  acquisition 
experience  (IG,  1992:12). 

Research  Questions 


1.  What  cost  management  competencies  are  of  value  to  intermediate  and  senior  level 
DoD  program  managers? 


A.  What  cost  management  competencies  are  perceived  to  be  important  by  program 
managers? 


B.  How  frequently  are  specified  cost  management  competencies  used  by  program 
managers? 


2.  Is  there  significant  variance  in  the  competencies  required  of  subgroups  of  this 
population? 

A.  Is  there  significant  variance  in  the  competencies  valued  by  those  in  intermediate 
and  senior  level  positions? 

B.  Is  there  significant  variance  in  the  competencies  valued  based  on  differences  in 
management  education? 

C.  Is  there  significant  variance  in  the  competencies  valued  by  those  working  with 
different  primary  program  activities? 

D.  Is  there  significant  variance  in  the  competencies  valued  by  those  working  in 
different  types  of  organizations? 


Scope 

Although  competencies  in  a  wide  spectrum  of  disciplines  such  as  engineering, 
contracting,  logistics,  and  cost  management  contribute  to  program  management  success, 
this  effort  focused  on  a  single  area  within  the  broader  discipline  of  program  management. 
In  particular,  this  research  focused  on  identifying  the  cost  management  competencies 
valued  by  intermediate  and  senior  level  military  program  managers  below  the  rank  of 
brigadier  general.  Competencies  were  also  evaluated  based  on  the  demographic 
subgroups  of  organization  type,  program  phase,  certification  required  for  the  position,  and 
the  type  of  management  education. 

Key  Terms 

The  following  key  terms  are  taken  from  DOD  5(XX).52-M,  Career  Development  Program 
For  Acquisition  Personnel. 
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Acquisition.  The  planning,  design,  development,  testing,  contracting, 
production,  introduction,  acquisition  logistics  suppon,  and  disposal  of  systems, 
equipment,  facilities,  supplies,  or  services  that  are  intended  for  use  in,  or  support 
of  military  missions. 

Acquisition  Corps.  A  subset  of  DoD  Con^nent’s  acquisition  workforce, 
composed  of  selected  military  and  civilian  personnel  in  grades  of  Lieutenant 
Commander,  Major,  General  Schedule  and/or  General  Manager  (GS/GM)  13  and 
above,  who  are  acquisition  professionals.  There  is  one  Acquisition  Corps  for 
each  Military  Department  and  one  for  all  the  other  DoD  Components  (including 
the  OSD  and  the  Defense  Agencies). 

Acquisition  Experience.  Experience  gained  while  assigned  to  an  acquisition 
position.  Also  includes  intern,  exchange,  education  or  training  with  industry,  and 
other  acquisition  developmental  assignments.  Includes  experience  in  DoD 
acquisition  positions  and  in  comparable  positions  outside  the  Department  of 
Defense. 

Acquisition  Organization.  An  organization,  including  its  subordinate  elements, 
whose  mission  includes  planning,  managing  and/or  executing  acquisition 
programs  which  are  governed  by  DoD  Directive  5000.1,  DoD  Instruction 
5000.2,  and  related  issuances. 

Acquisition  Positions.  Civilian  positions  and  military  billets  that  are  in  the  DoD 
acquisition  system,  have  acquisition  duties,  and  fall  in  an  acquisition  position 
category  established  by  the  USD(A).  While  most  frequently  located  in 
organizations  having  an  acquisition  mission,  acquisition  positions  are  also  located 
in  management  headquarters  organizations,  management  headquaners  support 
organizations,  and  other  organizations. 

Acquisition  Program.  A  directed,  funded  effon  that  is  designed  to  provide  a 
new  or  improved  materiel  capability  in  respcxtse  to  a  validated  need. 

Acquisition  Workforce.  The  personnel  component  of  the  acquisition  system. 
The  acquisition  workforce  includes  permanent  civilian  employees  and  military 
members  who  occupy  acquisition  positions,  who  are  members  of  an  Acquisition 
Corps,  or  who  are  in  acquisition  development  programs. 

Certification.  A  process  through  which  it  is  determined  that  an  individual  meets 
aU  the  education,  training,  and  experience  standards  established  for  his  or  her 
acquisition  career  field  or  position,  or  fcx*  membership  in  an  Acquisition  Corps. 

Critical  Acquisition  Position.  Those  senior  positions  carrying  signiHcant 
responsibility,  primarily  involving  supervisory  or  management  duties,  in  the  DoD 
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acquisition  system.  Those  positions  are  designated  by  the  Secretary  of  Defense, 
based  on  the  recommendations  of  the  DoD  Component  Acquisition  Executives, 
and  include  any  acquisition  position  required  to  be  filled  by  an  employee  in  the 
grade  of  GS/GM  14  or  above,  or  military  grade  0-5,  or  above.  Also  specifically 
includes  all  the  Program  Executive  Officers  (PEOs),  the  Deputy  PEOs,  the  PMs 
and  the  Deputy  PMs  for  major  defense  acquisition  programs,  and  the  PMs  of 
significant  non-major  programs. 

Mandatory  DoD  Acquisition  Course.  A  course  of  study  that  has  been 
identified  by  the  USD(A)  as  meeting  an  established  DoD  education  and  training 
requirement.  These  courses  provide  a  common,  non-component- specific 
foundation  of  knowledge  for  each  acquisition  function.  Each  of  the  following 
courses  is  mandatory  within  one  or  more  career  programs  or  is  mandatory  to 
qualify  for  certain  assignments,  or  both: 

a.  Career-development  Mandatory  Course.  A  course  that  must  be  taken 
for  an  employee  to  be  certified  at  Level  I,  II  or  III  within  one  of  the  career 
fields. 

b.  Qualification  Mandatory  Course.  A  course  that  must  be  completed  for 
an  employee  to  be  eligible  to  perform  certain  duties  or  to  be  given  certain 
assignments. 

Program  Manager  (PM).  A  military  or  civilian  official  who  is  responsible  for 
managing  an  acquisition  program. 

Prt^ram  Office.  An  acquisition  office  with  the  mission  to  plan,  manage,  or 
execute  an  acquisition  program.  [Used  interchangeably  with  System  Program 
Office  (SPO)]. 

Summary 

Scandals  and  failures  in  defense  acquisition  such  as  those  encountered  on  the  Air 
Force  C-17  and  the  Navy  A- 12  programs  have  plagued  the  Department  of  Defense  for 
years,  resulting  in  negative  publicity  and  increased  scrutiny.  Numerous  review  committees 
have  attempted  to  improve  the  process  and  structure  of  the  defense  acquisition  system.  In 
addition  to  the  process  itself,  recent  efforts  have  also  focused  on  the  people  involved  in 
DoD  acquisition.  There  has  been  a  concerted  effort  to  improve  the  quality  of  the 
acquisition  workforce  through  legislation  such  as  DAWIA  and  service  programs  such  as 
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APDP.  Education  and  training  play  a  major  role  in  this  improvement  process.  In  order  to 
maximize  the  benefit  gained  for  the  time  and  money  invested  in  education  and  training,  it 
is  critical  to  focus  these  programs  on  the  tools  that  will  be  most  valuable  to  program 
managers  in  their  jobs.  This  thesis  was  an  attempt  to  address  the  cost  management 
competencies  required  of  Air  Force  program  managers. 

Chapter  2  documents  efforts  made  to  date  to  improve  the  acquisition  workforce 
and  to  identify  the  competencies  required  of  program  managers.  Studies  of  program 
management  skills,  traits,  and  competencies  are  discussed,  as  are  the  few  studies  that 
address  cost  management  directly.  Chapter  3  describes  the  development  of  the 
competency  model,  the  sampling  approach,  survey  instrument  development,  and  the  data 
analysis  procedures.  The  findings  and  results  of  the  analysis  are  presented  in  Chapter  4. 
Chapter  5  presents  the  conclusions  of  this  effort  and  recommendations  for  funher  study. 
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n.  Literature  Review 


Overview 

This  chapter  documents  the  findings  of  an  extensive  literature  review.  First,  a  brief 
description  of  the  unique  nature  of  DoD  program  management  is  given.  Then,  a  review  of 
general  management  and  project  management  models  is  presented  chronologically  in  two 
sections;  Private  Sector  Research  and  Government  Research.  This  review  of  applicable 
literature,  cost  management  models,  and  relevant  lessons  learned  from  previous  DoD 
acquisition  programs  lays  the  foundation  for  development  of  a  cost  management 
competency  model,  to  be  addressed  in  Chapter  3,  Methodology. 

The  Nature  of  DoD  Program  Management 

The  Project  Management  Institute  (PMI)  is  a  professional  organization  formed  to 
advance  “the  state-of-the-art  in  the  management  of  projects”  (Wideman,  1987 : 1- 1 ).  In  its 
Project  Management  Body  of  Knowledge  (PMBOK),  a  model  which  identifies  the  critical 
aspects  of  project  management,  PMI  defines  project  management  as,  “The  art  of  directing 
and  coordinating  human  and  material  resources  throughout  the  life  of  a  project  by  using 
modem  management  techniques  to  achieve  predetermined  objectives  of  scope,  cost,  time, 
quality,  and  participant  satisfaction”  (PMI,  1987). 

The  unique  nature  of  project  management  can  best  be  described  in  the  context 
of  the  characteristics  of  a  program.  Nicholas  presents  several  characteristics  of  a 
project.  A  project  involves  a  single,  definable  purpose,  end-product  or  result,  which 
is  usually  specified  in  terms  of  cost,  schedule,  and  performance  requirements. 
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Projects  require  skills  from  multiple  professions  and  organizations,  thus  they  cut 
across  organizational  lines.  Projects  are  also  temporary,  one-time  activities,  never  to 
be  exactly  repeated  again  (Nicholas,  1990:3-4). 


These  project  characteristics  may  be  applied  to  any  project,  and  they  distinguish 
the  project  management  environment  from  other  types  of  management  characterized  by 
on-going,  repetitive  activities.  The  PMBOK  further  distinguishes  traditional  management 
fipom  project  management  by  identifying  the  following  characteristics  of  managing  an 
established  on-going  enterprise: 


•  life  in  an  on-going  enterprise  is  relatively  simple  and  certain  for  extended 
periods  of  time. 

•  Relatively  large  quantities  of  goods  or  services  are  produced  per  given  time 
period. 

•  Tasks  are  generally  repetitive,  continuous  or  exhibit  substantial  similarity. 

•  Roles  and  relationships  are  well  understood,  having  developed  and  adjusted 
over  long  periods  of  time,  and 

•  The  work  environment  is  relatively  stable. 

None  of  these  are  true  in  a  project  environment.  In  a  sense  every  project  is 
uitique,  if  only  by  virtue  of  its  own  set  of  constraints,  although  indeed  there  may 
be  many  projects  of  a  similar  nature.  (Wideman,  1987:1-1) 


Despite  these  differences,  many  of  the  cost  management  tools  used  in  DoD 
program  management  are  also  used  throughout  industry.  However,  the  literature 
reviewed  for  this  research  is  presented  in  the  context  of  the  DoD  acquisition  environment, 
which  includes  not  only  the  characteristics  of  a  project  listed  above,  but  also  the  effects  of 
woridng  with  public  funding  in  a  socially  and  politically  charged  environment. 
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Private  Sector  Research 

Researchers  in  the  private  sector  have  conducted  studies  and  analyses  in  a  variety 
of  areas  related  to  program  management.  Several  relevant  studies  are  presented 
chronologically  below. 

Levinson 

Hany  Levinson  looked  at  personality  criteria  that  could  be  used  to  select  senior 
executives  (Levinson,  1980:113-120).  Levinson’s  work  was  not  directed  toward  the 
military  program  manager,  although  senior  managers  in  both  industry  and  DoD 
environments  may  rely  on  similar  skills.  Levinson  conducted  no  formal  research  to 
validate  the  components  in  his  model  and  in  fact  stated,  “I  make  no  claim  for  statistical 
validation  of  the  dimensions  or  that  the  scales  represent  equal  intervals  or  accurate 
measures”  (Levinson,  1980:119).  Levinson’s  work  was,  “a  way  of  calling  attention  to, 
and  examining  facets  or  dimensions  of  personality  that  relate  to  executive  success” 
(Levinson,  1980:  1 19).  Levinson’s  20  “Dimensions  of  Leaders’  Personalities”  are  listed  in 
Table  3. 
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Table  3.  Levinson’s  20  Dimensions  of  Leaders’  Personalities 


Thinking 

•  Llapacity  to  Abstract 

•  Tolerance  for  Ambiguity 

•  Intelligence 

•  Judgment 

Feelings*  Interrelationships 

•  Authority 

•  Activity 

•  Achievement 

•  Sensitivity 

•  Involvement 

•  Maturity 

•  Interdependence 

•  Articulateness 

•  Stamina  (Physical  and  Mental) 

•  Sense  of  Humor 

•  Adaptability 

Outward  Behavior 

•  Vision 

•  Perseverance 

•  Personal  Organization 

•  Integrity 

•  Social  Responsibility 

(Levinson,  1980:  113-120) 


Kerzner 

Another  study  emphasizing  the  educational  aspect  of  developing  project 
management  competencies  was  conducted  by  Dr.  Harold  Kerzner.  He  surveyed  392  PMI 
members  and  asked  them  to  rank,  in  order  of  preference,  courses  they  thought  were,  or 
would  be,  important  to  their  professional  development  (Kerzner,  1981:38).  The  results  of 
this  study  are  presented  in  Table  4. 
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Table  4.  Course  Selection  Rankings 


Course 

#  of  times  identifled 
out  of  177  responses 

Fundamentals  of  Project  Management 

160 

Planning  and  Control 

142 

Accounting  and  Finance 

139 

OrganizatitHial  Behavior 

122 

Systems  Management/Organizational  Theory 

121 

Law 

109 

Information  Systems 

109 

105 

PERT/CPM 

97 

Computers 

79 

Management  Science 

75 

Managerial  Economics 

71 

Government,  Management,  Environment 

70 

Production  Management 

68 

Statistics 

55 

Marketing 

55 

Multinational/Intemational  Trade 

45 

Quality  Control 

39 

(Kerzner,  1981:42) 


Table  4  indicates  the  perceived  relative  importance  of  education  and  training  in  various 
aspects  of  project  management  One  other  important  point  Kerzner  emphasizes  is  that 
most  of  the  managers  surveyed  felt  “that  project  managers  cannot  be  trained  in  an 
academic  environment  and  that  colleges  and  universities  should  provide  only  the  necessary 
tools  for  general  project  ma^iagement  and  leave  the  formal  training  to  the  companies 
themselves”  (Kerzner,  1981:43),  He  further  highlighted  the  need  for  courses  to  focus  on 
application,  not  theory  (Kerzner,  1981:44). 
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Thornberry  and  Others 

Thombeny’s  team  conducted  an  empirical  study  to  “examine  a  sample  of  project 
managers  to  detetmine  the  fundamental  skills  and  abilities  necessary  for  project 
management  success”  (Thornberry  et  al.,  1983:73).  His  sample  consisted  of  1 10  male 
project  managers  who  were  identified  by  their  supervisors  as  successful  project  managers 
(Thornberry  et  al.,  1983:74).  Thombeny’s  study  was  multi-faceted,  involving  two  weeks 
of  time  log  keeping,  personality  testing,  and  personal  interviewing  with  a  sample  of  the 
participants.  The  personal  interviews  involved  discussion  of  a  successful  and  a  failed 
project  management  incident  in  which  the  participant  was  involved.  The  16PF 
(personality  factors)  test  was  used  to  identify  common  personality  traits  (Thornberry  et  al., 
1983:74).  Table  5  shows  Thomberry’s  five  “core  dimensions”  of  program  management 

Table  5.  Thomberry’s  Core  Dimensions 

1.  Oral  Communications _ 

2.  Influencing  Skills _ 

3.  Intellectual  Capabilities _ 

4.  Ability  to  Handle  Stress _ 

5.  Works  Skills  (planning,  organizing,  delegation,  and  decision  making) 

(Thornberry  et  al.,  1983:73-76) 

Project  Management  Institute 

The  PMI PMBOK  (previously  introduced)  was  designed  to  be  a  “well-defined 
body  of  knowledge  (BOK)  that  can  be  studied  and  learned  through  formal  education” 
(Stuckenbruck,  1986:25).  Table  6  shows  the  PMBOK’s  detailed  cost  management  model, 
including  the  elements  PMI  believes  are  necessary  for  a  well-defined  body  of  knowledge. 
Table  14  illustrates  how  cost  management  fits  into  the  complete  PMBOK. 
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Table  6.  PMBOK  Cost  Management  Elements 
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(Georgas  and  Stelco,  1987:D-1) 


Einsiedel 


Albert  Einsiedel  developed  a  non-empirical  model  of  project  management  skills  or 
traits.  His  efforts  were  directed  at  uncovering  characteristics  which  make  project 
managers  successful  (Einsiedel,  1987:51).  Einsiedel’s  research  provides  a  valuable  insight: 
Not  aU  projects  are  equally  sensitive  to  the  efforts  of  the  leader  (Einsiedel,  1987:51).  He 
notes,  “The  importance  of  obtaining  a  proper  match  between  the  individual’s 
characteristics  and  the  role  requirements  is  directly  related  to  the  degree  to  which  the 
project  is  leader  sensitive.”  (Einsiedel,  1987:54).  Einsiedel  presents  the  following  five 
“Characteristics  of  an  Effective  Project  Leader^’  (Einsiedel,  1987:53-4): 

1.  Credibility 

2.  Creative  Problem-solver 

3.  Tolerance  for  Ambiguity 

4.  Flexible  Management  Style 

5.  Effective  Communication  Skills 

Posner 

Barry  Posner  conducted  an  empirical  study  of  project  managers.  He  received 
almost  1400  response  items  from  287  respondents  to  two  open-ended  questions: 

1)  What  factors  or  variables  are  most  likely  to  cause  you  problems  in  managing  a 
project?;  and 

2)  What  personal  characteristics,  traits,  or  skills  do  “above  average”  project 
managers  use  (or  use  better  than  their  peers)?  (Posner,  1987: 107) 

Participants  included  189  men  and  98  women  and  ranged  from  22  to  60  years  old. 

Posner  categorized  the  responses  to  both  questions  as  shown  in  Table  7  (the  percentage  of 

respondents  identifying  the  particular  area  in  their  response  is  in  parenthesis). 
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Table  7.  Posner’s  Problems  and  Skills 


_ Problems _ 

1.  Resources  Inadequate  (69%) _ 

2.  Meeting  (unrealistic)  deadlines  (67%) _ 

3.  Unclear  goals  /  direction  (63%) _ 

4.  Team  members  uncommitted  (59%) _ 

5.  Insufficient  planning  (56%) _ 

6.  Breakdowns  in  communications  (54%) 

7.  Changes  in  goals  /  resources  (42%) _ 

8.  Conflicts  between  departments 

or  functions  (35%) _ 

_ Skills _ 

1.  Communication  (84%) 

•  listening  and  persuading _ 

2.  Organizational  Skills  (75%) 

•  planning 

•  goal  setting 

•  analyzing _ 

3.  Team  Building  Skills  (72%) 

•  empathy 

•  motivation 

•  esprit  de  corps _ 

4.  Leadership  Skills  (72%) 

•  sets  example 

•  energetic 

•  vision  (big  picture) 

•  delegates 

•  positive _ 

5.  Coping  Skills  (stress  management)  (59%) 

•  flexibility 

•  creativity 

•  patience 

•  persistence _ 

6.  Technological  Skills  (46%) 

•  experience 

•  project  knowledge _ 

(Posner,  1987:108) 
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There  are  some  similarities  between  the  most  common  problems  and  the  most 
valued  skills.  For  example,  breakdowns  in  communication  make  communication  skills 
valuable,  and  unclear  goals,  insufficient  planning,  and  changes  in  goals  make 
organizational  skills  critical. 

Cadbury-Schweppes 

Cadbury-Schweppes  conducted  an  extensive  research  program  involving  program 
manager  competencies.  Utilizing  a  variety  of  approaches,  Cadbury  developed  a 
“competency  language”  describing  “Dimensions  of  Management”  (Glaze,  1989:72-78). 
Cadbury’s  use  of  assessment  centers  results  in  more  emphasis  on  the  requirements  of 
individual  positions  than  on  the  profession  as  a  whole  (Glaze,  1989:72-78).  In  this 
context,  precise  definitions  of  the  requirements  fra:  individual  positions  run  contrary  to  the 
definition  of  a  generic  set  of  program  manager  competencies,  unless  that  set  is  so  general 
that  it  provides  little  basis  for  differentiation  between  applicants. 

Cadbury-Schweppes  developed  a  six  cluster  model  for  dimensions  of  management 
that  they  use  to  profile  both  positions  and  managers  (Glaze,  1989:73, 76-77).  They 
started  with  40  behavioral  competency  definitions  that  had  been  developed  by  a 
consultant.  CJadbury-Schweppes  modified  this  list  to  arrive  at  a  list  of  50  competencies. 
These  50  competencies  were  then  divided  into  six  categories  (clusters)  as  indicated  in 
Table  8.  Although  no  formal  validation  has  been  accomplished  on  dtis  particular 
competency  nnodel,  Cadbury-Schweppes  clairt^  validation  through  years  of  successfully 
using  the  model  as  a  tool  in  personnel  offices  for  profiling  employees,  assessing  potential 
performance,  and  establishing  training  needs  fm*  individual  positions. 


Table  8.  Cadbury-Schweppes  Model 


1.  Strategy _ 

2.  Drive _ 

3.  Analysis _ 

4.  Implementation _ 

5.  Personal  Factors _ 

6.  Influence _ 

•  reladonships _ 

•  persuasion _ 

•  leadership _ 

•  followership _ 

(Glaze,  1989:76) 


Sargent  and  Stupak 

Sargent  and  Stupak  saw  a  trend  in  program  management  toward  inclusion  of 
characterisdcs  and  techniques  regarded  as  more  feminine.  In  doing  so,  they  move  away 
from  traditional  masculine  characteristics  and  propose  the  set  of  seven  androgynous 
program  manager  competencies  shown  in  Table  9  (Sargent  and  Stupak,  1989:29-35). 


Table  9.  Androgynous  Competencies 


1.  Technical  Competence _ 

2.  Problem-Solving  Competence _ 

3.  Self-Awareness _ 

4.  Interpersonal _ 

5.  Team  Leadership  and  Membership _ 

6.  Entrepreneurial _ 

7.  Leadership _ 

(Sargent  and  Stupak,  1989:34-5) 


Williams  and  Currey 

Williams’  and  Cuncy’s  study  was  directed  toward  logistics  managers.  They 
developed  their  model  through  analysis  of  the  contents  of  the  want  ads  in  an  Australian 
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newspaper.  They  looked  for  the  most  commonly  noted  skills  or  attributes  in  those  adds. 
The  study  included  a  sample  of  70  ads  gathered  over  a  15  month  period  (Williams  and 
Currey,  1990:370).  Their  presumption  that  companies  write  their  valued  management 
skills  into  the  ads  seems  logical.  However,  the  authors  note  that  “There  may  be  attributes 
not  advertised  for  reasons  of  economy  of  expression,  racial  bias,  gender  bias  and  other 
unstated  biases”  (Williams  and  Cuney,  1990:369).  Williams’  and  Cuney’s  “desired 
attributes  for  logistics  managers”  are  shown  in  Table  10. 

Table  10.  Logistics  Manager  Attributes 


Skills 

•  Communication 

•  Personal  Relations  /  Leadership 

•  Factual  Logistic  Knowledge 

•  Money  Flow  Management  Evaluation 

•  Computer  Literacy 

Experience 

•  Large  Organizations 

•  Proven  Track  Record 

•  At  least  5  years  Management  Experience 

Qualities  /  Qualifications 

•  Motivated  and  Creative 

•  Pleasant,  Couneous  and  Professional 

•  Contributes  to  Firm’s  Objectives 

•  Independent  and  Accountable 

•  Tertiary  Qualifications 

(Williams  and  Currey,  1990:375) 


Tire  elements  of  this  model  are  similar  to  some  of  the  other  models  presented  here, 
but  computer  literacy  and  independence  are  not  frequently  encountered  in  the  other 
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models.  Williams  and  Currey  indirectly  address  one  issue  of  this  study  in  considering  what 

their  model  implies  for  education  (Williams  and  Currey,  1990:377).  The  authors  note: 

If  the  results  of  this  study  are  any  guide,  logistics  syllabi  must  take  a  holistic 
perspective.  The  study  not  only  identified  the  importance  of  the  traditional 
topics  of  materials  management,  inventory  management,  warehousing  and 
transport,  but  also  identified  the  importance  of  related  subjects  of  accounting, 
communication,  psychology  and  computing.  (Williams  and  Currey,  1990:377) 

Williams  and  Currey  reference  another  relevant  study  conducted  by  Poist  and 
Mattingly  which  found  that  “post  bachelor-degreed  logistics  personnel  attached  a  higher 
importance  to  quantitative  techniques  than  personnel  with  less  education  (Williams  and 
Currey,  1990:376).  A  similar  phenomena  may  be  present  in  project  management  circles 
wherein  certain  competencies  are  valued  more  by  personnel  with  more  education  and 
training. 

Pettersen 

Pettersen  derived  a  set  of  predictors  for  program  manager  success.  These 
predictors  (Table  1 1)  were  based  solely  on  the  results  of  other  studies  as  documented  in 
over  60  publications  (Pettersen,  1991:21-2). 
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Table  11.  Pettersen’s  Predictors 


Other  Personal  Qualities 

•  Need  to  Achieve  and  Proactivity 

•  Self-Confidence,  Maturity,  Stability 

•  Loyalty,  Honesty,  Integrity 

•  Tolerance  Towards  Ambiguity 

•  Interest  in  the  Job 

Interpersonal  Relations 

•  Oral  Communication 

•  Influence 

•  Ascendancy 

Supervision  and  Project  Team  Management 

•  Delegation  of  Responsibilities 

•  Team  Structuring 

•  Consideration  Towards  Team 

•  Team  Member  Development 

•  Teamwork,  Flexibility,  Cooperation 

•  Resolving  Conflicts 

Administration 

•  Planning  and  Organization 

•  Control 

•  Strategy  and  Organizational  Know-How 

•  Specialized  Knowledge 

Problem  Solving 

•  Problem  Analysis 

•  Judgment  and  Practical  Sense 

•  Decisiveness 

(Pettersen,  1991:22) 


Thamhain 

Thamhain  investigated  “the  effectiveness  of  various  approaches  to  project 
management  training  and  development  as  perceived  by  managers  in  the  field”  (Thamhain, 
1991:39).  His  conclusions  were  based  on  interviews  and  records  reviews  of  220  project 
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managers  from  18  companies  and  business  divisions,  having  annual  operating  budgets 
ranging  from  $2.5  million  to  $15  billion  (Thamhain,  1991:39). 

Thamhain  identified  leadershipAnterpersonal,  technical,  and  administrative  skills  as 
die  primary  areas  of  concern  for  developing  project  management  training  programs.  He 
then  asked  project  managers  to  indicate,  from  the  12  methods  below,  which  were  most 
desirable  for  developing  overall  competence  in  project  management 

1.  Experiential  Learning 

2.  Observing  Management  Practice 

3.  Formal  On-The-Job  Training 

4.  Literature  Reading 

5.  Coaching  By  Upper  Management 

6.  Seminars  and  Workshops 

7.  Formal  Courses  (Degree) 

8.  Consulting  (Internal  and  External) 

9.  Professional  Caiferences 

10.  Special  Work  Groups 

1 1.  Formal  Courses  (Continuing  Education) 

12.  Job  Rotation 

Then,  the  project  managers  were  asked  to  indicate  the  training  methods’  approximate 
contribution  to  professional  development,  relative  to  other  methods  (Thamhain,  1991:44). 
Figure  1  shows  the  significance  of  various  training  methods  as  perceived  by  the  project 
managers  in  Thamhain’s  research.  Figures  are  rounded  to  the  nearest  whole  percent. 
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(Thamhain,  1991:41) 


Figure  1.  Contribution  of  Skill  Development  Methods 

Kanungo  and  Misra 

Kanungo  and  Misra  attempt  to  distinguish  between  managerial  skills  and 
competencies.  They  state  that  competencies  refer  to  the  mental  capabilities  that  lead  to 
successful  adaptation  to  the  real-world  context,  especially  for  non-routine  tasks  in  a 
volatile  environment.  These  capabilities  are  the  inner  resources  that  managers  possess. 
Skills  on  the  other  hand,  are  needed  for  tasks  that  are  routine  or  programmed  in  a  stable 
environment.  Without  accompanying  competencies,  these  technical  and  routine  task 
related  skills  will  remain  dormant  (Kanungo  and  Misra,  1992:1322). 

Kanungo  and  Misra  reject  the  idea  of  a  single  set  of  competencies  when  they 
reference  past  studies  showing  that  both  academic  standing  and  past  job  performance  are 
poor  predictors  of  future  performance  because  of  variance  in  managerial  positions  and 
difficulty  in  specifying  requirements  (Kanungo  and  Misra,  1992:1313).  Kanungo  and 
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Misra  provide  a  set  of  three  general  categories  of  competencies:  affective,  intellectual, 
and  action-oriented.  A  more  detailed  break-out  of  the  components  of  these  three 
competencies  is  provided  below  (Kanungo  and  Misra,  1992:1323-29). 

1.  Affective  Competence 

-  Controlling  primitive  terminal  reactions  in  situations  that  produce 
strong  emotions 

-  Developing  equanimity  and  problem  orientation 

-  Delay  of  gratification 

-  High  proactive  involvement,  enthusiasm,  interest,  and  commitment  to 
meeting  challenges  of  life 

2.  Intellectual  Competence 

-  Intellectual  competence  to  solve  problems 

-  Self-reflection  for  strengthening  self-efficacy  belief 

3.  Action-Oriented  Competence 

-  Task-related  action  orientation  with  regard  to  goal  and  plan  development 
and  use  of  feedback 

-  People-related  action  orientation 


Kanungo  and  Misra’s  model  has  not  been  validated  by  research,  but  rather  it  is  a  review 
and  critique  of  relevant  literature. 


Government  Research 

Smythe  and  McMuilan 

An  effort  by  Smythe  and  McMuilan  compared  program  manager  skills  required  in 
various  phases  of  the  acquisition  process.  The  results  of  personal  interviews  with  24 
System  Program  Directors  for  major  programs  indicated  that  different  skills  are  required 
throughout  the  life  of  a  program  (Smythe  and  McMuilan,  1975:41).  This  result  casts 
doubt  (Ml  the  validity  of  using  a  single  set  of  competencies  for  all  positions  since  program 
managers  may  be  involved  with  a  program  during  more  than  one  phase.  Smythe  and 
McMuilan  specifically  identified  a  distinction  between  the  skills  necessary  during  the 
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development  phase  and  those  required  during  production/deployment  (Smythe  and 
McMuUan,  1975). 

Gadefcen 

Gadeken  directed  a  contractor’s  study,  sponsored  by  the  Defense  Systems 
Management  College  (DSMC),  which  identified  16  competencies  based  on  interviews  with 
56  program  managers  from  the  Army,  Navy,  and  Air  Force  acquisition  commands 
(Gadeken,  1989a;  Gadeken,  1989b;  Gadeken  et  al.,  1990;  DSMC,  1990).  ‘Two  groups  of 
program  managers  were  selected  for  interviews:  a  group  of  outstanding  performers  and  a 
contrasting  group  of  effective  (or  more  typical)  performers”  (Gadeken,  1989:22b).  These 
interviews  were  used  to  generate  critical  situations  involving  program  managers.  These 
critical  situations  were  later  analyzed  to  identify  specific  characteristics  which  were  then 
categorized  to  make  up  the  preliminary  competency  model. 

The  preliminary  model  was  validated  through  a  foUow-on  survey  questionnaire 
distributed  to  more  than  5(X)  acquisition  professionals.  Gadeken  identified  two  alternative 
structures  of  program  manager  competencies,  which  are  presented  in  Tables  12  and  13. 


32 


Table  12.  Gadeken’s  Program  Management  Model  (Form  1) 


Leadership 

1.  Sense  of  Ownership  /  Mission 

2.  Long-Term  Perspective 

3.  Assertiveness 

4.  Managerial  Orientation 

Problem  Solving 

10.  Proactive  Information  Gathering 

11.  Strategic  Inquiry 

12.  Systematic  Thinking 

13.  Interpersonal  Assessment 

Achievement 

5.  Focus  on  Excellence 

6.  Results  Orientation 

7.  Innovativeness  /  Initiative 

8.  Action  Oriented 

9.  Optimizing 

bifluence 

14.  Political  Awareness 

15.  Relationship  Development 

16.  Strategic  Influence 

(DSMC,  1990:E-l) 


Table  13.  Gadeken’s  Program  Management  Model  (Form  2) 


Managing  the  External  Environment 

Managing  for  Enhanced  Performance 

1.  Sense  of  Ownership  /  Mission 

10.  Long-Term  Perspective 

2.  Political  Awareness 

11.  Focus  on  Excellence 

3.  Relationship  Development 

12.  Innovativeness  /  Initiative 

4.  Strategic  Influence 

13.  Optimizing 

5.  Interpersonal  Assessment 

6.  Assertiveness 

14.  Systematic  Thinking 

Managing  the  Internal  Environment 

Proactivity 

7.  Managerial  Orientation 

15.  Action  Oriented 

8.  Results  Orientation 

9.  Critical  Inquiry 

16.  Proactive  Information  Gathering 

(Gadeken  and  others,  1990:27) 


Best  and  Kobylarz 

The  earlier  work  of  the  Project  Management  Institute  to  establish  a  Project 
Management  Body  of  Knowledge  was  extended  to  the  DoD  in  an  Air  Force  Institute  of 
Technology  graduate  thesis  by  Best  and  Kobylarz  (Best  and  Kobylarz,  1991).  Their  scope 
statement  includes  a  reference  to  defining, "...  the  knowledge  necessary  for  a  DoD  project 
manager  to  perform  effectively,”  but  their  survey  was  conducted  only  with  program 


33 


directors  and  deputies  who  are  typically  the  grade  of  colonel/GM-15,  or  higher  (Best  and 
Kobylarz,  1991:3).  Since  there  are  project  managers  at  many  levels  in  DoD,  a  survey  of 
senior  level  personnel  in  one  organization  type  may  not  provide  information  that  can  be 
generalized  to  the  entire  population  of  project  managers. 

Their  methodology  involved  ranking  the  elements  of  their  model.  This  method 
provides  only  relative  rankings  and  provides  no  basis  for  assuming  that  all  elements  in  the 
model  are  required.  Best  and  Kobylarz  integrated  the  PMI PMBOK  with  the 
requirements  of  the  Defense  Systems  Management  College  (DSMC)  Program  Manager 
Course  to  arrive  at  a  Defense  Body  of  Knowledge  (DBOK)  (Best  and  Kobylarz,  1991). 
The  elements  of  these  three  models  are  shown  in  Table  14: 
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Table  14.  PMBOK,  DSMC  PMC,  DBOK  Model  Comparisons 


PMBOK 

DSMC  PMC 

DBOK 

Scope 

Managerial  Development 

Time 

Management  Techniques 

Communication  Sc 
Information 

Leadership  &  Personal 
Skills 

Human  Resources 

Principles  of  Program 
Management 

(Quality 

(^ality  Management 

Risk 

Risk  Management 

Cost 

Cost/Schedule  Control 

Cost  Management 

Contractor  Finance 

Funds  Management 

Procurement 

Contract  Management 

Contract  Management 

Systems  Engineering 

Systems  Engineering 

Test  &  Evaluation 
Management 

Test  &  Evaluation 
Management 

Logistics 

Logistics 

Manufacturing  Management 

Manufacturing  Management 

Software 

Software  Management 

International  Program 
Management 

Aerospace  &  Defense 
Management 

Defense  Acquisition  Policy 
&  Environment 

(Best  and  Kobylarz,  1991:38) 


Acquisition  Management  Functional  Board 

The  Acquisition  Management  Functional  Board  (AMFB)  is  a  high  level,  joint 
service  board  with  representation  from  all  three  services  and  from  the  Defense  Systems 
Management  College  (DSMC).  Its  goal  was  to  “obtain  a  practical  set  of  competencies  for 
the  [acquisition  management]  career  field”  (AMFB,  1993: 1).  The  board  identified 
competencies  it  felt  required  formal  education  and  notes  that  “the  list  of  validated  program 
manager  competencies  requiring  formal  school  can  now  be  used  to  audit  existing  courses” 
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(AMFB,  1993:2).  Its  model  is  structured  by  primary  activity  with  a  series  of  competencies 
under  each  activity  (AMFB,  1993:1).  The  model  grouped  a  total  of  129  competencies 
into  the  12  areas  shown  in  Table  13. 


Table  15.  AMFB  Program  Management  Model 


7.  Managerial  Development 

2.  Contract  Finance 

8.  Manufacturing  Management 

3.  Contract  Management 

9.  Program  Management 

4.  Cost/Schedule  Control 

10.  Software  Management 

3.  Fiscal  Management 

11.  Systems  Engineering 

6.  Logistics  Support 

12.  Test  and  Evaluation 

(AMFB,  1993) 


Summarv 

The  various  studies  and  models  presented  here  show  the  diversity  of  methods  used 
to  present  the  competencies  required  of  program  managers.  Approaches  have  varied  from 
the  consideration  of  general  personality  traits  and  skills  to  identification  of  specific 
knowledge  and  ability  requirements.  A  significant  amount  of  the  research  on  program 
management  has  focused  on  personality-type  characteristics,  while  specialized  areas  like 
cost  management  have  received  little  attention.  Specifically,  the  area  of  cost  management 
is  addressed  cmly  by  the  AMFB,  Best  and  Kobylarz,  PMBOK,  and  Williams  and  Currey 
models  in  this  literature  review. 

The  subjective  personality-type  competencies  like  leadership  and  communication 
which  comprise  the  majority  of  this  literature  are  certainly  relevant  to  characterizing 
general  traits  of  successful  program  managers,  but  they  do  little  towards  identifying 
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competencies  in  specific  program  management  areas  (like  cost  management)  in  sufficient 
detail  to  develop  education  and  training  programs. 

Of  the  literature  reviewed,  the  PMBOK  and  AMFB  models  provide  the  greatest 
level  of  detail  in  cost  management.  However,  these  models  do  not  identify  the  perceived 
value  of  each  cost  management  competency  or  any  variance  among  subgroups  of  the 
population  of  interest.  Air  Force  program  managers.  Without  this  capability,  the  research 
accomplished  to  date  is  of  limited  use  in  optimizing  education  and  training.  A  model 
identifying  which  individual  competencies  program  managers  actually  value  in  their  day- 
to-day  activities  would  provide  this  type  of  information.  The  model  developed  in  Chapter 
3  of  this  thesis  was  designed  to  support  this  purpose. 

The  method  of  acquiring  cost  management  competencies  is  not  the  focus  of  this 
research.  However,  it  is  a  logical  question  following  the  identification  of  the  competencies 
that  program  managers  rely  on.  Research  by  Kerzner  identified  the  perceived  relative 
importance  of  education  and  training  courses  in  various  aspects  of  project  management. 
Kerzner  and  Thamhain  both  emphasized  the  importance  of  acquiring  project  management 
skills  through  experience  and  special  programs  developed  by  individual  organizations  to 
meet  their  individual  needs.  These  studies  were  included  in  the  literature  review  to  begin 
to  bridge  the  research  of  what  competencies  are  valuable,  with  future  research  of  how  the 
DoD  should  develop  them. 
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in.  Methodology 

Overview 

A  goal  of  this  research  was  to  identify  cost  management  competencies  valuable  to 
military  program  managers.  In  order  to  meet  this  goal,  a  cost  management  competency 
model  was  developed  based  on  past  research.  While  the  model  was  based  on  cost 
management  research,  the  perceptions  of  program  managers  in  the  field  were  used  to 
evaluate  it.  Perceptions  were  gathered  through  a  mail  survey  of  682  intermediate  and 
senior  level  military  program  managers  in  Air  Force  Materiel  Command.  The  study  also 
evaluated  differences  in  perceptions  among  program  managers  from  different  organization 
types,  management  education  backgrounds,  primary  program  activities,  and  position 
levels. 

The  primary  focus  of  this  effort  was  determining  the  implications  the  results  have 
for  the  development  and  training  of  program  managers  in  the  area  of  cost  management. 
The  remainder  of  this  chapter  describes  the  development  of  the  competency  model,  the 
sampling  approach,  survey  insoument  development,  and  the  data  analysis  procedures. 

Model  Development 

Integration  of  the  available  material  into  a  concise  model  required  consideration  of 
the  relevance  and  level  of  indenture  of  the  topic  area  as  well  as  model  efficiency.  The  goal 
was  to  develop  a  broad-based  model  reflecting  the  most  important  cost  management 
competencies,  not  an  exhaustive  list  of  every  tool  and  technique  a  program  manager  might 
use.  The  top  level  structure  of  the  model  and  its  contents  is  described  below. 
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Model  Structure 


The  structure  of  the  model  was  based  on  the  cost  management  section  of  the 
Project  Management  Body  of  Knowledge  (PMBOK)  (PMI,  1987).  In  order  to  simplify 
the  structure,  the  fourth  PMBOK  category  (Cost  Applications)  was  eliminated. 

Important  competencies  from  this  section  were  integrated  into  other  categories.  The 
remaining  three  categories  represent  the  major  steps  in  the  cost  management  process.  The 
resulting  top-level  structure  of  the  model  was: 


Table  16.  Model  Structure 


Cost  Management 

Cost  Estimating  & 
Forecasting 

Cost  Budgeting 

Cost  Controls 

Model  Elements 

The  competency  topics  were  synthesized  from  the  contents  of  the  PMBOK  and  the 
weak  of  the  Acquisition  Management  Functional  Board  (AMFB).  The  PMBOK  contains 
a  number  of  cost  management  competency  areas  grouped  into  the  four  categories 
discussed  above  (PMI,  1987).  The  AMFB  competency  model  identified  a  number  of 
competencies  in  the  cost  management  area  and  specified  a  desired  level  of  learning 
required  at  each  of  the  three  APDP  certification  levels  (AMFB,  1993).  The  AMFB 
categivized  the  level  of  learning  using  a  six  stage  taxonomy  of  learning  developed  by 
Bloom  (1956).  Table  17  illustrates  the  elements  of  Bloom’s  taxonomy,  reflecting 
increasing  levels  of  learning. 
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Table  17.  Bloom’sTaxonomy  of  Learning 


Knowledge 


Comprehension 


Application 


Analysis 


Synthesis 


Evaluation 


(Bloom,  1956:  18) 


The  competency  statements  in  this  study’s  model  also  considered  the  level  of 
learning  required,  focusing  on  whether  the  program  manager  needed  to  be  able  to  perform 
a  particular  task  or  understand  the  results  of  the  task.  Many  cost  management  topics  in 
the  model  thus  consist  of  two  competencies,  a  be  able  to  conqretency  and  an  understand 
competency.  The  levels  of  learning  used  in  this  study,  while  potentially  including 
characteristics  of  more  than  one  level,  are  roughly  analogous  to  Bloom’s  comprehension 
and  application  levels. 

The  competency  topic  areas  were  identiBed  through  consideration  of  the  elements 
in  the  AMFB  and  PMBOK  models.  The  relevance  and  level  of  indenture  of  the  topic 
areas  was  considered  in  an  attempt  to  balance  the  need  to  span  the  range  of  important  cost 
management  topics  with  the  desire  for  a  concise  model.  Redundant  topics  or  topics  less 
directly  tied  to  cost  management  were  not  included.  The  competencies  were  organized 
within  the  structure  deRned  above.  The  result  is  the  47  element  model  shown  in  Table  18. 
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Table  18.  Cost  Management  Competency  Model 


1.0  Cost  Estimating  &  Forecasting 

1.1  Understand  current  economic  conditions  and  their  impact  on  defense  contractors 

1.2  Be  able  to  evaluate  contractor  financial  health  and  viability  using  financial  statements 

1.3  Understand  the  results  of  financial  statement  analysis  conducted  to  evaluate  contractor  financial  health 

1.4  Be  able  to  apply  time  value  of  money  techniques  such  as  return  on  investment  (ROI),  net  present  value 
(NPV),  internal  rate  of  return  (IRR),  and  DCF 

1.S  Understand  the  impact  of  the  time  value  of  money  on  financing  and  budgeting 

1.6  Understand  how  contractors  secure  funds  to  support  ongoing  projects,  plant  improvements,  and  new 
product  development 

1.7  Be  able  to  develop  a  cost  estimate  using  appropriate  methods  (e.g.  parameuic,  analogy,  grass  roots) 

1.8  Understand  cost  estimates  developed  using  appropriate  methods  (e.g.  parametric,  analogy,  grass  roots) 

1.9  Be  able  to  use  statistical  analysis  methods  such  as  range  analysis  and  confidence  intervals  to 
characterize  uncertainty  associated  with  cost 

1.10  Understand  the  implications  of  uncertainty  associated  with  cost  estimates 

1.11  Be  able  to  use  software  tools  to  support  cost  estimation,  analysis,  and  presentation 

1.12  Understand  the  products  of  cost  management  software  tools 

1.13  Understand  how  contractors  apply  management  reserve  to  respond  to  contingencies  over  the  duration 
of  a  contract 

1.14  Be  able  to  apply  inflation  factors  to  program  costs  and  funding 

1.15  Understand  the  impact  of  inflation  on  program  costs  and  funding 

1.16  Understand  the  impact  of  production  rate  and  quantity  decisions  on  program  cost 

1.17  Understand  the  impact  of  budget  <  uts  on  unit  marginal  cost 

1.18  Understand  cost  elements  such  as  direct  labor,  direct  materials,  general  &  administrative,  profit,  and 
overhead 

1.19  Understand  ways  to  characterize  costs  such  as  fixed/variable  and  recurring/non-recurring 

1.20  Be  able  to  do  break-even  analysis  using  the  concepts  of  fixed  and  variable  costs 

1.21  Understand  the  cost  concepts  of  reasonableness,  allocability,  and  allowability 

1.22  Be  able  to  apply  learning  curve  techniques  to  analyze  production  costs 

1.23  Understand  the  impact  of  learning  curves  on  production  costs 

2.0  Cost  Budgeting 

2.1  Be  able  to  develop  a  Work  Breakdown  Suiicture  (WBS)  that  describes  the  entire  work  effort 

2.2  Understand  the  uses  of  a  WBS  for  cost  management 

2.3  Understand  the  role  the  organization  plays  in  the  Planning,  Programming,  and  Budgeting  System 
(PPBS) 

2.4  Understand  the  congressional  budgeting  and  appropriations  processes 

2.5  Understand  the  flow  of  funds  through  the  expenditure  categories  of  commiunents,  obligations,  and 
expenditures 

2.6  Be  able  to  distribute  the  cost  of  work  packages  across  the  time  horizon  to  develop  a  performance 
measurement  baseline  (PMB) 

2.7  Understand  the  composition  of  a  PMB 

2.8  Be  able  to  evaluate  conuactor  cost  accounting  and  control  systems 

2.9  Understand  contractor  cost  accounting  and  control  systems 

2.10  Be  able  to  select  the  appropriate  contract  type  for  a  project 

2.1 1  Understand  the  cost  implications  of  alternative  contract  types  and  pricing  mechanisms 

2.12  Understand  the  impact  of  the  political  environment  on  acquisition  management 

2.13  Be  able  to  generate  congressionally  required  reports  such  as  the  SAR 

2.14  Understand  congressional  reporting  requirements 
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_ 3.0  Cost  Controls _ 

3.1  Understand  the  legal  and  regulatory  requirements  for  cost  and  schedule  control  systems  such  ns 

cyscsc _ 

32  Understand  the  impact  of  changes  in  scope  on  the  cost  of  defense  contracts _ 

3.3  UndCTStand  the  concept  of  earned  value  and  methods  for  calculating  it _ 

3.4  Be  able  to  estimate  earned  value  using  methods  such  as  weighted  milestones  and  percent  complete 


3.10  Be  able  to  develop  corrective  actions  to  counter  unfavorable  program  variances 


An  important  step  in  the  data  collection  and  analysis  process  was  determining  the 
population  and  target  sample  for  administering  the  survey  to  evaluate  the  cost 
management  competencies  in  the  model.  Careful  definition  of  these  groups  sets  the 
foundation  for  developing  the  survey  instruments  and  interpreting  the  results. 

Population 

The  cost  management  competencies  required  of  military  program  managers  was 
the  focus  of  this  research.  In  particular,  the  study  focused  on  the  perceptions  of 
intermediate  and  senior  level  program  managers.  The  effort  focused  on  Air  Force 
Materiel  Command  (AFMC)  which  is  responsible  for  most  Air  Force  acquisition 
programs. 

The  population  and  sample  groups  were  identified  using  the  program  management 
certification  level  required  for  the  position.  Each  program  management  position  is  labeled 


42 


with  the  desired  certification  level  of  the  program  manager  who  should  fill  it  Intermediate 
and  senior  level  program  managers  fill  Level  II  and  Level  IE  positions,  respectively.  The 
Acquisition  Management  Resource  Development  Team  at  AFMC  (AFMC/XRMA) 
maintains  a  database  that  identifies  the  person  filling  each  program  management  position  in 
AFMC.  The  database  contained  a  total  of  8 16  military  program  management  positions 
coded  Level  n  or  Level  m  as  of  Febmary  1994.  Since  junior  program  managers  are  not 
likely  to  have  a  significant  experience  base  upon  which  to  form  opinions  regarding  the 
competencies  in  the  model,  they  were  excluded  from  the  population.  General  officers 
were  excluded  to  avoid  impacting  their  schedules.  The  following  statement  identifies  the 
populaticm. 

Population:  Military  personnel  below  the  grade  of  brigadier  general  occupying 

Level  II  and  Level  in  program  management  positions  in  AFMC. 

Target  Sample 

The  entire  population  was  contacted  with  the  exception  of  those  not  located  on 
military  bases.  The  remaining  group,  which  included  682  of  the  816  program  managers  in 
the  population,  became  the  target  sample  of  this  research  effort. 

Two  survey  instrument  versions  were  used  in  order  to  minimize  the  time  required 
to  complete  the  survey.  Feedback  from  an  instrument  pre-test  indicated  the  need  to  split 
the  questions  into  two  instmments,  as  discussed  below.  The  first  instrument  focused  on 
perceived  importance  of  the  competencies  while  the  second  focused  on  frequency  of 
use.  Each  instrument  version  was  distributed  to  341  program  managers.  Since  the 
research  involved  analysis  of  subgroups  within  the  population,  a  large  target  sample  size 
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ensured  representative  coverage  of  each  category  (organization  type,  position  certification 
level,  management  education,  and  primary  program  activity). 

To  ensure  balance  between  the  frequency  and  importance  groups,  the  Level  II  and 
Level  in  position  lists  were  sorted  by  military  rank  and  by  base  of  assignment.  Half  of 
each  list  was  then  assigned  to  the  frequency  instrument  version  and  half  to  the  importance 
version  on  an  alternating  basis.  This  approach  resulted  in  balanced  sample  groups  in  terms 
of  rank  and  location  of  respondents.  The  survey  was  distributed  to  the  following  groups 
in  terms  of  position  level  and  instrument  version: 

Table  19.  Target  Sample  Size  by  Instrument  Version  and  Position  Level 


GrouD 

Version  /  Level 

Number 

1 

Frequencv  /  Level  n 

245 

2 

Frequencv  /  Level  HI 

96 

3 

Importance  /  Level  11 

245 

4 

Importance  /  Level  HI 

96 

Sample 

The  survey  was  disnibuted  by  mail  to  these  sample  groups.  Responses  were 
requested  within  4  weeks  of  the  distribution  date.  A  total  of  330  program  managers 
returned  usable  data  for  a  response  rate  of  48.4%.  The  responses  of  these  330  managers 
constitute  the  sample  for  data  analysis  purposes.  The  instruments  were  distributed  to  four 
groups  of  program  managers  based  on  the  APDP  certification  level  associated  with  the 
position  and  on  which  of  the  two  instrument  versions  (Frequency  or  Importance)  they 
received.  Figure  2  shows  the  number  of  respondents  from  each  group. 
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Nunnber  of  Rospondonte 


Figure  2.  Number  of  Respondents  by  Instrument  Version 


Biographical  questions  provided  demographic  information  on  the  respondents. 
Figures  3, 4,  and  5  show  the  percentage  of  resp'^idents  from  each  category. 
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Figure  3.  Percentage  of  Respondents  by  Organization  Type 

Figure  3  highlights  the  concentration  of  acquisition  positions  in  the  SPOs. 
Approximately  5%  of  the  respondents  did  not  fit  into  one  of  the  common  organization 
types  provided  in  the  instruments. 


GraduatB  Osg-m 
andOSMCPMC 
45% 


Graduate 

Mmagemern  Degee 
28% 


OSMCRogam 
Manager  Oourae 
15% 


Figure  4.  Percentage  of  Respondents  by  Management  Education 
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The  largest  group  of  respondents  had  completed  both  a  graduate  management 
degree  and  the  DSMC  Program  Manager  Course.  In  all,  88%  of  respondents  had 
completed  some  form  of  graduate  management  education. 


Oevelopirantal 

Systems 

50% 


Figure  5.  Percentage  of  Respondents  by  Primary  Program  Activity 


Consistent  with  the  large  number  of  respondents  from  SPOs,  most  respondents 
work  with  developmental  and  mature  systems.  Only  15%  of  the  respondents  work  with 
the  early  stages  of  the  process  dealing  with  concepts  and  technologies. 
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Figure  6.  Number  of  Respondents  by  Position  Certification  Level 


The  sample  reflects  the  larger  number  of  Level  n  program  managers  in  the 
population.  Approximately  two-thirds  of  the  respondents  were  serving  in  Level  n 
program  management  positions.  Intemiediate  level  program  managers  thus  had  a  greater 
contribution  to  overall  results  than  senior  level  managers.  Figure  6  also  shows  that  the 
response  rates  for  Level  n  and  Level  III  program  managers  were  comparable  (46.3  %  and 
53.6  %  respectively). 

Instrument  Development 

The  first  step  in  the  development  of  the  survey  instruments  was  to  determine  how 
to  assess  the  value  of  each  competency.  Once  this  was  accomplished,  the  response  scales 
and  question  structure  could  be  specified.  The  instruments  were  structtired  to  be  both 
valid  and  reliable.  A  pre-test  provided  feedback  on  the  draft  instrument. 
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Frequency  and  Importance 

The  primary  attribute  of  interest  in  this  research  was  the  value  of  each  competency. 
Quantifying  value  is  a  difficult  task.  One  approach  is  to  solicit  only  the  frequency  of  use 
of  each  competency.  Unfortunately,  there  could  be  competencies  that,  although  rarely 
used,  are  absolutely  critical  to  the  success  of  the  program.  For  example,  a  program  might 
only  develop  a  work  breakdown  stracture  (WBS)  once,  but  the  WBS  is  the  base  upon 
which  the  entire  program  is  built.  Therefore,  frequency  alone  is  not  enough. 

The  impOTtance  of  a  competency  is  another  indicator  of  its  value.  This  measure 
accounts  for  the  value  of  competencies  such  as  WBS  development.  Although  infrequently 
used,  such  competencies  could  be  extremely  important  The  two  measures,  frequency  and 
importance,  were  integrated  through  use  of  a  decision  rule  to  assess  the  value  of  each 
competency. 

Response  Scales 

The  primary  data  in  this  research  was  the  reported  frequency  of  use  and  perceived 
importance  of  competencies  scored  on  descriptive  ordinal  scales.  The  critical  decisions  in 
defining  a  scale  include  how  many  responses  to  include  and  what  descriptors  to  use. 
Meister  cites  studies  noting  that: 

...  there  is  apparently  no  gain  in  reliability  if  one  increases  the  number  of 
categories  ftom  5  to  9,  but  reliability  drops  with  3  (too  gross)  or  more  than  7 
(too  fine) ...  There  appears  to  be  little  utility  in  having  more  than  5  scale 
categories.  (Meister,  1985:326) 

A  choice  must  be  made  regarding  how  extreme  the  ends  of  the  scale  should  be 
(Meister,  1985:382).  This  choice  has  a  direct  impact  on  the  number  of  categories  in  the 


49 


scale.  For  example,  in  developing  a  scale  for  frequency  one  could  use  very 
frequently/infrequently  or  always/never  as  the  extremes,  thus  leaving  only  three  other 
alternatives.  Neither  of  the  response  scales  used  for  this  research  included  absolute 
values,  so  five  value  response  scales  were  sufficient  A  sixth  response,  don’t  know, 
accommodated  respondents  unfamiliar  with  particular  competencies. 

Frequency 

The  frequency  response  scale  for  this  effort  employed  a  series  of  time  intervals 
between  uses  of  the  competency.  Table  20  presents  the  response  scale. 


Table  20.  Frequency  Response  Scale 


1 

2 

3 

4 

5 

6 

Annually 
or  less 

Quanerly 

Monthly 

Weekly 

Daily 

Don’t 

Know 

Importance 

The  importance  response  scale  was  a  series  of  ordinal  descriptors  reflecting 
incremental  increases  in  perceived  importance.  Table  21  shows  the  scale  used  in  the 
instruments  (Meister,  1985:382-9). 


Table  21.  Importance  Response  Scale 


1 

2 

3 

4 

5 

6 

Not 

Important 

Slighdy 

Important 

Important 

Very 

Important 

Extremely 

Important 

Don’t 

Know 

SO 


Pre«Test  Instrument  Structure 

A  survey  instrument  was  developed  using  the  competencies  and  response  scales 
described  above.  This  instrument  was  pre-tested  on  a  group  of  program  management 
students  and  modified  based  on  their  feedback.  The  pre-test  instrument  contained  three 
sections.  The  first  section  requested  biographical  information  regarding  the  following 
areas: 


-  Civilian  or  Military  Status 

-  Organization  Type 

-  System  Program  Office,  Headquarters,  Logistics  Center,  Laboratory 

-  Management  Education 

-  Completion  of  DSMC  Program  Manager  Course  or  a  Graduate 
Management  Degree 

-  Primary  Activity  /  Program  Phase 

-  Concepts,  Technologies,  Developmental  Systems,  Mature  Systems 


The  second  section  of  the  pre-test  survey  evaluated  perceived  importance  of  each 
competency  in  the  model  based  on  a  five-point  scale  of  suitable  ordinal  descriptors.  The 
third  section  evaluated  frequency  of  use  of  each  competency  on  a  similar  five-point  scale. 
The  instrument  contained  a  total  of  100  questions. 


Pre-Test 

The  initial  instrument  was  pre-tested  on  a  group  of  12  graduate  program 
management  students.  Specific  areas  of  interest  for  the  pre-test  were  the  time  to  complete 
the  instrument  as  well  as  clarity  of  the  instructions,  response  scales,  and  competency 
statements.  Written  comments  on  the  instrument  were  solicited  via  open-ended  questions 
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on  a  feedback  sheet  provided  after  the  pre-test  respondents  completed  the  survey 
(Meister,  1985:387). 

The  pre-test  resulted  in  significant  changes  to  the  instrument.  The  average  time  to 
complete  the  survey  was  21.6  minutes.  The  respondents  were  asked  to  mark  the 
computer  data  sheet  at  five  minute  increments  throughout  the  pre-test.  This  informadon 
was  used  to  determine  the  question  response  rate  (Table  22). 


Table  22.  Pre-Test  (^estion  Response  Rate 


Time  (Minutes) 

Average  Questions  Completed 

10 

45 

15 

67 

20 

89 

25 

98 

There  were  several  verbal  and  six  written  comments  that  the  instrument  was  too 
long.  Although  the  instrument  only  took  22  minutes  on  average  to  complete,  with  100 
questions,  it  was  perceived  to  be  too  long.  Based  on  this  feedback,  the  instrument  was 
divided  into  the  frequency  and  importance  versions  mentioned  above.  Since  each 
respondent  addressed  either  frequency  or  importance  instead  of  both,  this  essentially 
halved  the  number  of  questions  fm*  the  final  instrument.  Surveying  both  civilian  and 
military  employees  for  this  research  was  administratively  prohibitive.  As  a  result,  the 
survey  focused  on  military  personnel  and  the  question  regarding  civilian  or  military  status 
was  deleted. 


Seven  of  the  twelve  respondents  missed  the  change  in  scales  fiom  in^rtance  to 
firequency.  The  final  instrument  included  large,  bold  section  headings  to  clearly  delineate 
all  transitions.  Six  respondents  suggested  that  they  needed  a  choice  between  annually  and 
monthly  on  the  frequency  response  scale.  The  nxxlified  frequency  scale  combined  two 
terms  and  added  a  quarterly  option  as  indicated  in  Table  23. 


Table  23.  Frequency  Response  Scale  Comparison 


Initial  Instrument 

Final  Instrument 

Less  than  once  per  year 

Annually  or  less 

Annually 

Quarterly 

Weekly 

Weekly 

Daily 

E^y 

Don’t  Know 

The  final  instrument  thus  consisted  of  two  versions,  frequency  and  importance. 
Each  version  contained  56  questions.  Both  instruments  are  provided  for  reference  in 
Appendix  A. 

Survey  Methods 

There  are  a  variety  of  approaches  to  developing  and  distributing  mail  surveys  to 
achieve  a  high  response  rate.  The  key  is  to  develop  an  integrated  cover  letter,  instrument, 
and  distribution  process  that  motivates  the  respondent  to  complete  the  instrument 
(Dillman,  1978).  Several  actions  were  taken  to  elicit  a  credible  response  rate  for  this 
survey. 
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The  cover  letter  accompanying  the  survey  instruments  in  this  effon  incorporated 
several  recommendations  from  the  literature  designed  to  motivate  the  respondents.  The 
letter  identified  the  research  with  a  known  organization,  the  Air  Force  staff  at  the 
Pentagon.  Mr.  Teddy  Houston,  the  Associate  Director  of  Acquisition  for  Career 
Management  under  the  Assistant  Secretary  of  the  Air  Force  (Acquisition),  signed  the 
cover  letter.  Mr.  Houston  is  responsible  for  aafuisition  professional  development  in  the 
Air  Force.  His  letter  stressed  the  value  of  the  research  in  improving  training  and  education 
programs;  noted  that  respondents  were  selected  based  on  their  experience,  education,  and 
training;  and  requested  their  support  (Dillman,  1978:13,18,162-69).  A  time-to-complete 
estimate  was  noted  in  the  cover  letter  emphasizing  the  minimal  amount  of  time  required 
(Dillman,  1978:14-15,18,54-55).  All  respondents  were  assured  anonymity  and  assured 
that  no  adverse  action  would  be  taken  for  not  responding  (Dillman,  1978: 170;  Meister, 
1985:384).  A  copy  of  this  cover  letter  is  provided  with  the  survey  instmments  in 
Appendix  A. 

The  questions  were  grouped  into  four  sections:  biographical,  frequency  or 
importance,  reliability  checks,  and  open-ended  questions.  The  biographical  questions 
formed  the  basis  for  subgroup  analysis.  Grouping  the  frequency  and  importance 
questions  allowed  placement  of  a  single  scale  at  the  top  of  the  page  that  could  be  applied 
to  ail  questions  on  that  page.  The  final  section  asked  if  any  significant  competencies  were 
omitted,  asked  respondents  to  identify  their  most  important  and  most  frequently  used 
comperencies,  and  provided  space  for  any  additional  comments. 
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Validity 

Validity  and  reliability  are  concepts  that  reflect  the  quality  and  performance  of  an 
instrument.  Validity  is,  the  extent  to  which  a  test  measures  what  we  actually  wish  to 
measure”  (Emory  and  Cooper,  1991:179).  External  validity  refers  to  the  generalizability 
of  the  results,  while  internal  validity  refers  to  ability  to  measure  constructs  (Emory  and 
Cooper,  1991:179-80). 

Internal  Validity 

This  effort  examined  two  constmcts:  the  perceived  importance  and  frequency  of 
usage  of  a  set  of  cost  management  competencies.  Choice  of  suitable  descriptor  scales  is 
one  way  to  improve  internal  validity.  A  good  descriptor  scale  directiy  addresses  the 
constructs  of  interest  and  adequately  reflects  variation  in  those  constmcts. 

Confusion  regarding  die  intended  meaning  of  the  competency  could  be  a  source  of 
error,  (^estions  may  be  too  vague  or  too  precise,  and  researchers  must  consider  how 
much  knowledge  they  assume  respondents  have  (Dillman,  1978:99-101, 1 12).  One  way 
to  stmcture  the  instruments  was  to  describe  each  competency  in  a  single  sentence.  The 
limitation  of  this  approach  is  that  it  was  likely  that  some  statements  would  not  be 
understood  by  the  respondents. 

An  alternative  approach  is  to  provide  detailed  competency  descriptions  in  the 
question  or  at  the  end  of  the  instmment.  However,  if  a  respondent  required  a  definition  of 
the  competency,  it  is  not  likely  that  it  is  important  to  or  frequently  used  by  that  manager. 
The  additional  material  would  have  made  the  instrument  longer  and  more  complex. 
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The  chosen  approach  used  simple  and  brief  competency  statements.  Examples  of 
specific  methods  within  a  competency  were  provided  to  clarify  the  topic.  For  example, 
the  cost  estimation  competencies  mention  the  parametric,  analogy,  and  grass  roots 
methods.  To  accomnaodate  those  who  did  not  understand  a  particular  statement  or  were 
not  familiar  with  the  competency,  both  the  frequency  and  importance  response  scales 
included  a  Don’t  Know  option. 

Spyridakis  discusses  eight  threats  to  internal  validity  (1992:612-13).  Two  of  these 
were  particularly  relevant  to  this  rese  xh.  Tie  first  was  bias  resulting  from  the  sample 
selection  process  (Spyridakis,  1992:612).  The  use  of  two  survey  instruments  versions 
mandated  an  effort  to  ensure  that  the  sample  groups  completing  each  version  were  similar. 
If  the  two  groups  did  not  reflect  the  same  population,  the  resulting  data  would  be  of 
questionable  value. 

The  two  instrument  versions  differed  in  the  aspect  of  each  competency  that  they 
measured.  One  instrument  focused  on  perceived  importance,  while  the  other  focused  on 
frequency  of  use.  The  solution  to  the  problem  was  for  each  group  to  evaluate  five 
competencies  in  terms  of  the  alternate  aspect.  Thus,  those  completing  the  importance 
version  of  the  instrument  evaluated  all  47  competencies  in  terms  of  importance  and  five  in 
terms  of  frequency  of  use.  The  responses  for  those  five  questions  were  then  compared  to 
the  same  five  questions  for  the  group  completing  the  frequency  version  of  the  instrument. 
All  ten  questions  compare  the  same  two  groups  resulting  in  ten  independent  assessments 
of  equality  between  the  groups. 
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A  Kruskal-Wallis  test  on  each  question  determined  whether  the  population 
distributions  were  identical.  The  p-values  presented  in  Table  24  represent  the  smallest 
level  of  significance  that  would  result  in  rejectitm  of  the  hypothesis  that  the  response 
distributions  are  the  same  (Devore,  1991:315). 


Table  24.  Group  Kruskal-Wallis  Tests 


Question 

1 

.8958 

.0016 

2 

.7994 

.0573 

3 

.9564 

.2060 

4 

.1289 

.0174 

5 

.0013 

In  four  out  of  ten  cases,  the  results  indicate  that  the  two  sample  groups  reflect 
different  populations  (a  =  .05).  This  provides  some  indication  that  the  groups  differed. 
However,  no  obvious  reason  for  the  differences  was  evident  The  change  in  question 
format  from  frequency  to  importance  (or  the  reverse)  may  have  contributed.  This  area  of 
concern  was  noted  in  the  pre-test.  Although  the  instrument  was  revised  to  highlight  the 
change,  the  transition  may  still  have  been  confusing.  The  p-values  appear  to  be  lower  for 
the  group  focusing  on  perceived  importance.  This  could  indicate  that  the  transition  from 
importance  to  frequency  may  have  caused  more  difficulty. 

Another  way  to  illustrate  the  potential  differences  between  the  frequency  and 
importance  groups  is  to  look  at  the  ranking  of  the  five  competencies  used  for  comparison. 
The  full  set  of  47  competencies  can  be  ranked  based  on  the  proportion  answering  four  or 
five  on  the  response  scales.  The  response  data  for  the  other  group  can  then  be  substituted 
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for  the  five  competencies  used  for  comparison.  Since  the  response  data  may  differ,  the 
relative  rank  of  these  competencies  within  the  full  set  of  47  may  change.  Appendix  G 
illustrates  the  change  in  rank  that  would  result  if  the  data  from  the  alternate  group  were 
used.  In  eight  of  the  ten  cases,  the  shift  is  three  places  or  less. 

The  second  threat  is  that  the  subjects’  expectations  will  affect  their  responses 
(Spyridakis,  1992:613).  The  survey  package  did  not  provide  any  significant  details  about 
the  research  that  might  bias  responses.  The  cover  letter  accompanying  the  package  did 
not  list  any  specific  goals  for  the  research  other  than  exploration  of  this  area  (Dillman, 
1978:  167).  A  similar  problem  is  that  respondents’  results  may  be  biased  if  they  feel 
pressure  to  respond  in  a  certain  way  (Emory  and  Cooper,  1991:178).  This  effect  was 
minimized  by  assuring  anonymity  to  all  respondents  and  not  mandating  participadon. 

External  Validity 

External  validity  deals  with  generalizability.  This  research  effort  involved  a  variety 
of  program  managers  within  AFMC.  However,  the  target  sample  excluded  program 
managers  who  were  not  located  on  military  bases.  For  example,  program  managers 
serving  in  Defense  Plant  Representative  Offices  were  not  contacted.  Those  not  contacted 
constituted  approximately  16%  of  the  population.  It  is  possible  that  this  omission 
induced  some  bias  in  the  results  if  managers  in  these  positions  value  different  cost 
management  competencies  than  those  in  the  target  sample.  However,  approximately  84% 
of  the  population  received  a  version  of  the  survey  instrument,  which  provides  a  substantial 
representation  of  the  population  across  a  variety  of  organization  types. 
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Reliability 

Reliability  refers  to  consistency  in  the  results  of  an  instrument  (Emory  and  Cooper, 
1991:185).  Evaluation  of  instrument  reliability  often  involves  repeated  testing  and 
comparison  of  results  (Emory  and  Cooper,  1991:180).  Due  to  the  senior  level  of  portions 
of  the  population  and  the  necessity  to  limit  the  impact  of  the  research  on  Air  Force 
persrmnel,  it  was  not  practical  to  retest  One  competency  question  in  each  instmment 
version  was  duplicated  allowing  a  measure  of  internal  consistency  for  each  respondent 
Spearman’s  rank  correlation  was  used  to  evaluate  correlation  in  the  paired  data.  Table  25 
provides  the  values  for  Spearman’s  rho. 


Table  25.  Internal  Consistency  Correlation 


Frequency  Instrumrat 

Importance  Instrument 

Correlation 

.7281 

.5672 

These  values  are  somewhat  lower  than  expected,  but  may  have  resulted  from 
changing  perceptions  as  the  respondent  completed  the  instrument.  Another  method  for 
measuring  internal  consistency  is  to  evaluate  the  magnitude  of  the  difference  in  the  two 
answers  for  each  respondent.  Table  26  provides  a  frequency  distribution  for  these 
differences. 
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Table  26.  Internal  Consistency  Differences 


Difference 

Frequency  Instrument 

Importance  Instrument 

0 

69.0  % 

46.4  % 

1 

21.9  % 

42.8  % 

2 

9.0% 

9.0% 

3 

0% 

1.8% 

Less  than  1 1%  of  the  respondents  differed  by  more  than  1.  Given  the  subjective 
nature  of  the  response  scales,  these  results  provide  a  fairly  good  indicadon  that  the 
respondents  were  internally  consistent 

Data  Analysis 

Approach 

The  data  analysis  approach  is  dependent  upon  the  structure  of  the  response  data. 
Data  analysis  for  this  study  involved  two  primary  tasks.  Answering  the  first  research 
question  required  evaluation  of  the  proposed  model  using  data  from  the  entire  sample. 

This  evaluation  was  accomplished  by  considering  both  frequency  and  importance  rankings 
individually.  A  decision  rule,  discussed  below,  merged  the  two  concepts  into  a  single 
measure  of  value  for  the  competencies.  The  second  research  question  required  analyzing 
differences  in  the  responses  among  subgroups  of  the  population.  The  Kruskal- Wallis  test 
was  used  to  identify  differences  in  the  responses  between  those  from  different  organization 
types,  program  activities,  position  level,  and  management  education  background. 


Response  Data  Structure 

Respondents  marked  their  answers  on  a  machine  readable  form.  The  forms  were 
then  machine  scanned,  producing  a  computer  file  of  the  response  data,  which  was  used  for 
subsequent  data  analysis.  It  was  important  to  consider  whether  data  from  these  scales  was 
ordinal  or  interval.  Ordinal  data  has,  “Order  but  no  distance  or  unique  origin”  (Emory, 
1991:172).  Interval  data,  on  the  other  hand,  has  order  and  a  constant  distance  between 
successive  points  on  the  scale.  Ordinal  data  precludes  using  statistical  procedures  such  as 
determination  of  mean  and  variance,  regression  analysis,  and  standard  analysis  of  variance. 
These  procedures  require  at  least  interval  level  data  (Conover,  1980:66).  Meister  presents 
results  from  a  1976  study  by  Dyer  that  studied  frequency  descriptor  lists  (Meister, 
1985:383).  The  results  indicate  that  data  points  on  the  frequency  descriptor  scale  studied 
were  not  evenly  spaced  and  therefore  represented  ordinal  data  at  best. 

The  importance  scale  was  a  list  of  descriptors.  There  was  no  basis  for  an 
assumption  of  equal  spacing  between  options,  so  the  data  was  considered  ordinal.  The 
E>on’t  Know  option  cannot  be  easily  ordered  with  the  other  responses.  As  a  result,  the 
proportion  of  respondents  answering  Don’t  Know  was  noted  for  each  competency,  but 
these  responses  were  not  included  in  the  analysis.  Data  analysis  with  ordinal  data  uses  the 
median  as  a  measure  of  central  tendency.  Conover  discusses  tools  such  as  the  Kruskal- 
Wallis  test  which  allow  non-parametric  analysis  of  variance  (Conover,  1980).  The 
Kruskal-Wallis  test  is  discussed  in  the  data  analysis  section . 
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Research  Question  1  Analysis 

The  model  was  first  evaluated  by  analyzing  the  frequency  and  importance  data 
individually.  The  median  response  value  for  each  competency  was  calculated,  as  was  the 
proportion  of  respondents  answering  four  or  five.  In  tenns  of  frequency  of  use  this 
corresponds  to  using  the  competency  Weekly  or  Daily.  For  the  importance  instrument, 
the  values  indicated  that  the  competency  was  considered  Very  Important  or  Extremely 
Important.  The  competencies  were  then  ranked  by  this  proportion  to  highlight  the  most 
important  and  most  frequently  used  competencies. 

The  frequency  and  importance  data  were  then  combined  by  applying  a  decision 
rule  to  the  data.  The  decision  rule  was  a  subjective  effort  to  integrate  frequency  and 
importance  into  a  single  measure  reflecting  the  value  of  each  competency.  Frequency  and 
importance  were  weighted  equally  in  the  decision  rule. 

Decision  Rule:  Any  competency  whose  median  values  for  importance  and 

frequency  sum  to  six  (6)  or  greater  is  considered  valuable. 

The  result  of  applying  the  decision  rule  was  a  list  of  competencies  that  were 
considered  valuable  by  program  managers  in  the  field. 

Research  Question  2  Analysis 

The  second  data  analysis  process  was  variance  analysis.  The  purpose  of  this 
analysis  was  to  evaluate  differences  in  the  competencies  required  of  program  managers  as 
a  function  of  organization  type,  program  management  education,  primary  program 
activity,  and  position  level.  The  elements  within  each  of  these  groups  are: 
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1.  OrganizaticMi  Type 

--  System  Program  Office,  Headquarters,  Logistics  Center,  Laboratory 

2.  Management  Education 

“  Completion  of  DSMC  Program  Manager  Course  and/or  a  Graduate 
Management  Degree 

3.  Primary  Activity  /  Program  Phase 

--  Concepts,  Technologies,  Developmental  Systems,  Mature  Systems 

4.  Position  Certtiication  Level 

--  Level  n.  Level  in 

Kruskal-Wallis 

The  subgroup  analysis  process  identified  differences  in  the  perceived  frequency  and 
impcHtance  of  each  competency  separately  using  the  Kruskal-Wallis  test  for  non- 
parametric  analysis  of  variance.  As  described  in  Conover  (1980:229-37),  this  test  allows 
for  comparison  of  sets  of  ordinal  data. 

The  hypotheses  for  a  Kruskal-Wallis  test  are: 

Ho:  All  of  the  k  population  distribution  functions  are  identical 

H*:  The  k  populations  do  not  have  identical  distributions 

Rejection  of  the  null  hypothesis  indicates  that  the  subgroup  responses  were  not 
identically  distributed.  This  indicates  that  the  subgroups  did  not  find  the  competency  in 
question  equally  important  or  that  they  did  not  use  it  equally  frequently.  The  populations 
for  the  Kruskal-Wallis  test  were  the  sample  subgroups  identified  above. 

The  Kruskal-Wallis  test  has  three  primary  assumptions  (Conover,  1980:230): 

1.  All  samples  are  random  samples 

2.  Independence  within  and  between  samples 

3.  The  measurement  scale  is  at  least  ordinal 

The  response  data  is  ordinal  as  discussed  above.  Consideration  of  a  single 
competency,  only  in  terms  of  frequency  or  importance,  results  in  independence  between 
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and  within  the  samples.  The  sample  assignment  process  described  above  should  have 
resulted  in  balanced  samples,  although  the  assignment  was  not  purely  random.  In 
addition,  those  not  on  military  bases  were  excluded  as  discussed  above.  Statistical 
assessments  were  made  as  part  of  the  analysis,  but  the  potential  impact  of  the  limitations 
should  be  noted.  Although  a  level  of  significance  of  .05  was  used  to  accept  or  reject  the 
null  hypothesis,  p-values  for  the  Kruskal- Wallis  tests  are  provided  in  appendices 
permitting  the  user  to  assess  the  significance  of  the  result  if  desired. 

Summary 

The  goal  of  this  research  was  to  improve  the  development  and  training  of  program 
managers  in  the  area  of  cost  management.  Cost  management  models  developed  by  the 
Project  Management  Institute  and  the  Acquisition  Management  Functional  Board  formed 
the  foundation  of  the  cost  management  competency  model  developed  in  this  study.  Two 
survey  instruments,  examining  the  perceived  importance  and  frequency  of  use  of  these 
competencies,  were  each  distributed  to  341  intermediate  and  senior  level  program 
managers.  Data  analysis  procedures  allow  specification  of  the  valuable  competencies  and 
analysis  of  variations  among  program  managers  with  different  management  education 
backgrounds,  organization  types,  primary  program  activities,  and  position  certification 
levels.  Chapter  IV  presents  the  results  obtained  by  implementing  this  methodology. 
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IV.  Findings  and  Analysis 


Introduction 

This  study  focused  on  two  major  research  objectives:  identifying  valuable  cost 
management  competencies  and  identifying  differences  in  responses  among  subgroups  of 
the  population.  The  perceptions  of  330  program  managers  were  used  to  answer  those 
questions.  The  remainder  of  this  chapter  answers  each  research  question,  in  turn,  and 
concludes  with  a  discussion  of  additional  competency  areas  identified  by  the  respondents. 

Research  Oue.stion  1 

Answering  the  tirst  research  question  involved  analysis  of  the  frequency  and 
importance  data  individually.  The  data  was  then  analyzed  collectively  through  the  use  of  a 
decision  rule  which  integrated  fiequency  and  importance  responses  into  a  single  measure 
of  value. 

Frequency 

The  frequency  data  was  ranked  based  on  the  percentage  of  respondents  using  each 
competency  Daily  or  Weekly.  Tables  27  and  28  present  the  10  most  frequently  and  10 
least  frequently  used  competencies  on  this  basis. 
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Table  27.  Most  Frequently  Used  Competencies 


Competency 

Percent  Using 
Daily  or  Weekly 

Understand  the  impact  of  the  political  environment  on  acquisition  management 

46.2 

Understand  the  flow  of  funds  through  the  expenditure  categories  of  commitments, 
obligations,  and  expenditures 

45.9 

Understand  the  impact  of  changes  in  scope  on  the  cost  of  defense  contracts 

42.3 

Understand  the  role  the  organization  plays  in  the  Planning.  Programming,  and 
Budgeting  System  (PPBS) 

37.8 

Understand  current  economic  conditions  and  their  impact  on  defense  contractors 

35.1 

IWft  fThl '  1 1  [t  ■  h  1  [*]  1  1  til  1  r<  (■«  1 1  n'»  1 1  ti  1 1)  ijiii  •)  1 M « t»! 

35.0 

Be  able  to  evaluate  contractor-recommended  corrective  actions  and  select  an 
apmopriate  course  of  action 

28.1 

Understand  cost  elements  such  as  direct  labor,  direct  materials,  general  & 
administrative,  profit,  and  overhead 

27.8 

Understand  ways  to  characterize  costs  such  as  fixed/variable  and  recurring/non- 
recurring 

27.6 

Understand  the  impact  of  inflation  on  program  costs  and  funding 

25.4 

Table  28.  Least  Frequently  Used  Competencies 


Competency 

Percent  Using 
Daily  or  Weekly 

Be  able  to  evaluate  contractor  financial  health  and  viability  using  financial 
statements 

4.5 

Be  able  to  analyze  contractor  reports  such  as  the  CPR  and  CSSR 

3.8 

Be  able  to  do  break-even  analysis  using  the  concepts  of  fixed  and  variable  costs 

3.4 

Be  able  to  distribute  the  cost  of  work  packages  across  the  time  horizon  to  develop 
a  performance  measurement  baseline 

3.4 

Be  able  to  evaluate  contractor  cost  accounting  and  control  systems 

2.6 

Be  able  to  apply  time  value  of  money  techniques  such  as  renim  on  invesunent 
(ROI),  net  present  value  (NPV),  internal  rate  of  return  (IRR),  and  declining  cash 
flows  (DCF) 

2.0 

Be  able  to  apply  learning  curve  techniques  to  analyze  production  costs 

2.0 

2.0 

Be  able  to  develop  a  Woik  Breakdown  Structure  (WBS)  that  describes  the  entire 
work  effort 

1.9 

Be  able  to  use  statistical  analysis  methods  such  as  range  analysis  and  confidence 
intervals  to  characterize  uncertainty  associated  with  cost 

0 
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It  is  interesting  to  note  that  9  of  the  10  nx^t  frequently  used  competencies  are 
understand  competencies,  while  all  of  the  least  frequently  used  competencies  are  be  able  to 
competencies.  The  single  be  able  to  competency  in  the  most  frequently  used  10  deals  with 
analyzing  contractor-recommended  corrective  actions.  This  indicates  that  program 
managers  must  understand  the  results  of  analysis  and  other  program  management  issues, 
but  are  rarely  required  to  complete  those  actions  themselves. 

Importance 

Ute  importance  data  was  ranked  based  on  the  proportion  of  respondents  who 
indicated  the  competency  was  Very  Important  or  Extremely  Important.  Tables  29  and  30 
present  the  10  most  important  and  10  least  important  competencies  based  on  this  ranking. 
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Table  29.  Most  Important  Competencies 


Competency 

Percent  Rating 
Very  or  Extremely 
Important 

Be  able  to  evaluate  contractor-recommended  corrective  actions  and  select  an 
appropriate  course  of  action 

85.1 

Understand  the  cost  implications  of  alternative  contract  types  and  pricing 
mechanisms 

81.5 

Understand  the  impact  of  changes  in  scope  on  the  cost  of  defense  contracts 

81.5 

Understand  the  role  the  organization  plays  in  the  Planning,  Programming,  and 
Budgeting  System  (PPBS) 

80.9 

Be  able  to  develop  corrective  actions  to  counter  unfavorable  program  variances 

79.8 

79.6 

Understand  the  impact  of  production  rate  and  quantity  decisions  on  program 
cost 

79.6 

Understand  the  flow  of  funds  through  the  expenditure  categories  of 
commitments,  obligations,  and  expenditures 

78.5 

Understand  cost  elements  such  as  direct  labor,  direct  materials,  general  & 
administrative,  prorit,  and  overhead 

75.6 

Understand  contractor  cost  reports  such  as  the  Cost  Performance  Report  (CPR) 
and  Cost  /  Schedule  Status  Report  (CSSR) 

73.9 

Table  30.  Least  Important  Competencies 


Competency  Percent  Rating 

Very  or  Extremely 
Important 


Understand  how  contractors  secure  funds  to  support  ongoing  projects,  plant  33.9 

improvements  and  new  product  development _ 


Be  able  to  develop  an  estimate  at  completion  (EAC)  based  on  data  presented  in  32.4 

contractor  performance  reports _ 


Be  able  to  apply  inflation  factors  to  program  costs  and  funding _ _ 32.1 


Be  able  to  do  break-even  analysis  using  the  concepts  of  fixed  and  variable  costs  | _ 28.5 


Be  able  to  distribute  the  cost  of  work  packages  across  the  time  horizon  to  28.2 

develop  a  performance  measurement  baseline _ 


Be  able  to  use  statistical  analysis  methods  such  as  range  analysis  and  25.9 

confidence  intervals  to  characterize  uncertainty  associated  with  cost _ 


Be  able  to  apply  learning  curve  techniques  to  analyze  production  costs _ _ 25.8 


Be  able  to  estimate  earned  vtdue  using  methods  such  as  weighted  milestone  24.3 

and  percent  complete _ 


Be  able  to  apply  time  value  of  money  techniques  such  as  return  on  investment  22.2 

(ROl),  net  present  value  (NPV),  internal  rate  of  return  (IRR),  and  declining 
cash  flows  (DCF) 


Be  able  to  use  software  tools  to  support  cost  estimation,  analysis,  and  22.0 

presentation _ 
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The  pattern  for  the  importance  data  is  similar  to  that  of  the  frequency  data.  While 
eight  of  the  ten  most  important  competencies  required  understanding,  nine  of  the  ten  least 
important  competencies  were  be  able  to  competencies. 

Decision  Rule 

A  decision  rule  provided  a  means  for  combining  the  concepts  of  frequency  and 
importance  into  a  single  measure  of  value  for  the  competencies.  The  decision  rule 
considered  the  competency  valuable  if  the  sum  of  its  median  values  for  frequency  and 
inq)ortance  totaled  six  (6)  or  greater.  Using  this  rule,  29  of  the  47  competencies  were 
considered  valuable.  A  complete  list  of  the  median  sums  is  provided  in  Appendix  D. 
Tables  31  through  33  group  the  valuable  competencies  into  three  tiers  by  the  sum  of  their 
medians.  Tier  one  competencies  had  a  median  sum  of  8,  tier  two  had  a  sum  of  7,  and  tier 
three  a  sum  of  six.  Higher  median  sums  indicates  competencies  that  are  both  important 
and  frequently  used  and  ate  thus  more  valuable. 

Table  31.  Tier  One  Competencies  (Most  Valuable) 

_ Competency _ 

Understand  the  role  the  organization  plays  in  the  Planning,  Programming,  and  Budgeting  System 

(PPBS) _ 

Be  able  to  evaluate  contractor-recommended  corrective  actions  and  select  an  appropriate  course  of  action 
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Table  32.  Tier  Two  Competencies 


Competency 

Understand  current  economic  conditions  and  their  impact  on  defense  contractors 

Understand  cost  estimates  developed  using  appropriate  methods  (e.g.  parametric,  analogy,  grass  roots) 

Understand  the  implications  of  uncertainty  associated  with  cost  estimates 

Understand  how  contractors  apply  management  reserve  to  respond  to  contingencies  over  the  duration  of 
a  contract 

Understand  cost  elements  such  as  direct  labor,  direct  materials,  general  &  administrative,  profit,  and 
overhead 

Understand  ways  to  characterize  costs  such  as  fixed/variable  and  recurring/non-recurring 

Umterstand  the  uses  of  a  WBS  for  cost  management 

Understand  the  congressional  budgeting  and  appropriations  process 

Understand  the  flow  of  funds  through  the  expenditure  categories  of  commiunents,  obligations,  and 
expenditures 

Understand  the  impact  of  the  political  environment  on  acquisition  management 

Understand  the  impact  of  changes  in  scope  on  the  cost  of  defense  contracts 

Understand  the  utilization  of  EAC  in  cost  management 

Understand  contractor  reports  such  as  the  Cost  Performance  Report  (CPR)  and  Cost  /  Schedule  Status 
Report  (CSSR) 

Be  able  to  analyze  contractor  reports  such  as  the  CPR  and  CSSR 

Be  able  to  develop  coirective  actions  to  counter  unfavorable  program  variances 


Table  33.  Tier  Three  Competencies 


Competency 

Understand  the  products  of  cost  management  software  tools 

Understand  the  impact  of  inflation  on  program  costs  and  funding 

Understand  the  impact  of  production  quantity  and  rate  decisions  on  program  cost 

Understand  the  impact  of  budget  cuts  on  unit  marginal  cost 

Understand  the  cost  concepts  of  reasonableness,  allocability,  and  allowability 

Understand  the  impact  of  learning  curves  on  production  costs 

Understand  the  composition  of  a  Performance  Measurement  Baseline  (PMB) 

Understand  contractor  cost  accounting  and  control  systems 

Understand  the  cost  implications  of  alternative  contract  types  and  pricing  mechanisms 

Understand  congressional  reporting  requirements 

Understand  the  legal  and  regulatory  requirements  for  cost  and  schedule  control  systems  such  as 

c/sese 

Understand  the  concept  of  earned  value  and  methods  for  calculating  it 
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Don’t  Know 


A  Don’t  Know  option  was  provided  on  both  the  frequency  and  importance 
response  scales  to  accommodate  those  who  did  not  understand,  or  were  not  familiar  with, 
the  competency.  Don’t  Know  responses  were  excluded  from  the  data  analysis  because 
they  could  not  be  ordered  in  the  response  scales.  Appendix  E  illustrates  the  percentage  of 
respondents  that  answered  Don’t  Know.  No  more  than  10%  of  the  responses  for  any 
individual  competency  were  Don’t  Know.  The  percentage  of  Don’t  Know  responses 
averaged  .09  %  for  the  importance  instrument  and  3.4  %  for  the  frequency  instrument. 
This  indicates  the  vast  majority  of  respondents  understood  the  comp  ;*ency  statements  in 
the  model.  The  difference  in  Don’t  Know  response  averages  between  the  two  instrument 
versions  could  indicate  that  respondents  had  a  more  difficult  time  conceptualizing  the 
competency  in  terms  of  frequency  of  use. 

Model  Categories 

The  cost  management  competency  model  developed  for  this  research  contained 
three  major  areas:  cost  estimating,  cost  budgeting,  and  cost  control.  Figure  7  illustrates 
the  percent  of  the  competencies  in  each  category  that  were  considered  valuable  per  the 
decision  mie. 
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Figure  7.  Valuable  Competencies  by  Model  Category 

This  data  indicates  that  more  of  the  cost  control  competencies  were  considered 
valuable  by  program  managers  according  to  the  decision  rule.  The  decision  rule  weighted 
importance  and  frequency  equally.  However,  the  proportions  answering  four  or  five  were 
higher  for  the  importance  instrument  This  could  result  in  some  bias  of  the  decision  rule 
results  in  favor  of  importance.  Since  cost  control  activities  are  probably  conducted  more 
frequently  than  estimation  or  budgeting,  a  revised  decision  rule  balancing  the  response 
data  might  make  the  cost  control  category  appear  even  more  valuable. 

Comprehension  vs.  Application 

The  cost  management  model  developed  for  this  study  includes  competencies  at 
both  the  comprehension  and  application  levels  of  learning.  There  are  18  Be  able  to 
competencies  and  29  Understand  competencies  in  the  full  model.  Figure  8  illustrates  the 
level  of  learning  required  for  the  valuable  competencies. 
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AppScation  Comprehension 

Level  of  Learning  Required 


Figure  8.  Level  of  Learning  Required  for  Valuable  Competencies 

This  result  clearly  indicates  that  more  of  the  comprehension  competencies  were 
found  to  be  valuable.  The  analysis  indicated  that  90%  of  the  comprehension  competencies 
were  valuable  while  only  17%  of  the  applicadon  competencies  were  valuable. 

Research  Question  2 

The  second  research  question  analyzed  differences  in  the  responses  between 
subgroups  of  the  sample.  The  Kruskal-Wallis  test  for  non-parametric  analysis  of  variance 
was  used  to  compare  the  response  distributions  of  the  subgroups.  A  detailed  discussion  of 
the  individual  groups  follows  an  overview  of  the  Kruskal-Wallis  results. 

Results  Overview 

The  analysis  process  for  each  group  consisted  of  98  Kruskal-Wallis  tests.  The 
tests  compared  the  subgroup  response  distributions  for  each  of  the  47  competencies  on  the 
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basis  of  frequency  of  use  or  perceived  importance.  For  example,  a  single  test  might 
evaluate  whether  program  managers  with  four  different  levels  of  management  education 
consider  a  given  competency  equally  important.  The  test  compares  the  response 
distributions  for  all  the  subgroups.  The  hypothesis  that  all  the  subgroups  share  the  same 
response  distribution  can  be  rejected  if  even  one  subgroup  has  a  different  response 
distributicni.  Appendix  F  contains  the  p-values  for  these  tests,  which  represent  the 
smallest  level  of  significance  that  would  result  in  rejection  of  the  hypothesis  that  the 
response  distributions  are  the  same  (Devore,  1991:315).  The  results  can  be  summarized 
by  looking  at  the  number  of  tests  that  identified  subgroup  response  differences.  Figure  9 
shows  the  number  of  competencies  with  response  distribution  differences  for  each 
demographic  group  and  for  both  instmment  versions. 


Figure  9.  Number  of  Kruskal- Wallis  Rejections 


74 


It  is  interesting  to  note  that  those  with  different  management  education 
backgrounds  responded  differently  to  only  one  of  47  competencies  on  each  instrument 
version.  Roughly  10%  of  the  Kruskal-Wallis  tests  identified  variations  in  the  perceived 
importance  or  frequency  of  use  of  competencies  on  the  basis  of  position  certification  level 
or  primary  program  activity.  Comparison  of  the  response  groups  from  different 
organization  types  revealed  an  interesting  dichotomy.  While  12  competencies  showed 
different  frequency  of  use,  none  differed  with  respect  to  perceived  importance. 

As  noted  in  the  methodology  chapter,  it  was  not  possible  to  identify  the  number  of 
respondents  in  each  subgroup  prior  to  distribution  of  the  survey  instruments.  Tables  34 
through  37  show  the  number  of  respondents  in  the  subgroups  when  stratified  based  on 
management  education,  OTganization  type,  primary  program  activity,  and  position 
certification  level. 


Table  34.  Management  Education  Respondents 


1  Instrument 

None 

Graduate  Degree 

DSMC  PMC 

Both 

Total 

17 

41 

25 

70 

153 

21 

46 

23 

70 

160 

Table  35.  Organization  Type  Respondents 


I  Instrument 

SPO 

Lab 

ALC 

Other 

Total 

123 

1 

11 

13 

7 

155 

119 

5 

12 

14 

9 

159 
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Table  36.  Primaiy  Pittgram  Activity  Respondents 


Instrument 

Concepts 

Technologies 

Developmental 

Systems 

Total 

5 

12 

77 

46 

140 

7 

19 

66 

54 

146 

Table  37.  Position  Level  Respondents 


1  Instrument 

Level  n 

Level  m 

Total 

108 

54 

162 

119 

49 

168 

The  number  of  respondents  assigned  to  specific  subgroups  varied  significantly. 

The  sample  size  in  several  of  these  subgroups  was  quite  small.  In  some  cases,  there  may 
only  be  a  few  members  in  the  entire  population.  While  it  is  still  possible  to  conduct  the 
Kruskal-Wallis  test,  the  small  sample  sizes  for  those  subgroups  may  not  adequately 
represent  the  population.  In  these  cases,  the  small  sample  can  bias  the  Kruskal-Wallis  test. 
The  subgroup  response  distributions  could  falsely  appear  equal  or  an  actual  difference 
might  not  be  identified. 

Management  Education 

There  were  only  two  competencies  for  which  the  perceived  importance  or 
frequency  of  use  varied  with  different  education  backgrounds.  The  fact  that  only  two 
differences  were  identified  in  98  tests  (47  frequency  and  47  importance)  indicates  that  the 
level  of  management  education  did  not  significantly  impact  either  the  perceived 
importance  or  frequency  of  use  of  the  competencies  in  the  model.  This  contradicts  the 
hypothesis  that  those  with  more  management  education  might  find  more  or  different 
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ccKnpetencies  valuable  because  of  their  increased  exposure  to  management  techniques  and 
tools.  Tables  38  and  39  show  the  rejected  competencies  and  the  median  values  for  the 
subgroups. 


Table  38.  Frequency  Differences  Based  on  Management  Education 


Subgroup  I 

Median  I 

Competency 

None 

Graduate 

Degree 

DSMC 

PMC 

Both 

Understand  cost  estimates  developed  using 
appropriate  methods  (e.g.  parametric,  analogy,  grass 
roots) 

Quarterly 

Quarterly 

Monthly 

Monthly 

Table  39.  Importance  Differences  Based  on  Management  Education 


Subgroup  Median 

Competency 

None 

Graduate 

Degree 

DSMC 

PMC 

Both 

Understand  the  impact  of  learning  curves  on 
tnoducdon  costs 

Very 

Important 

Important 

■Bi 

msm 

Organization  Type 

As  discussed  above,  some  subgroups  had  small  sample  sizes.  In  terms  of 
organization  type,  both  the  laboratory  and  other  subgroups  had  sample  sizes  of  less  than 
10  for  both  instruments.  Due  to  the  small  sample  size  in  these  two  subgroups,  and  the 
difficulty  in  drawing  meaningful  conclusions  about  the  other  subgroup,  they  were  excluded 
from  the  organization  type  Kruskal- Wallis  tests. 

There  was  an  interesting  disparity  between  the  two  instrument  versions  in  the 
number  of  differences  identified  based  on  organization  type.  The  results  indicated  that  for 
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12  competencies,  the  frequency  of  use  distribution  was  not  the  same  for  all  subgroups. 
However,  no  differences  were  identified  in  the  perceived  importance  of  the  competencies. 
This  indicates  that  while  the  competencies  are  considered  equally  important  in  all 
organizations,  they  are  not  equally  frequentiy  used.  Table  40  shows  the  competencies  not 
having  the  same  distribution. 


Table  40.  Frequency  Differences  Based  On  Organization  Type 


Subgroup  Median 

Competency 

SPO 

HQ 

ALC 

Understand  current  economic  conditioiis  and  their  impact 
on  defense  contractors 

Monthly 

Quarterly 

Quarterly 

Be  able  to  apply  time  value  of  money  techniques  such  as 
return  on  investment  (ROI),  net  present  value  (NPV), 
internal  rate  of  return  (IRR),  and  declining  cash  flows 
(DCF) 

Annually 
or  Less 

Annually  or 
Less 

Annually  or 
Less/ 
Quarterly 

Understand  the  impact  of  inflation  on  program  costs  and 
funding 

Quarterly 

1 

Monthly 

Weekly 

Understand  the  impact  of  production  rate  and  quantity 
decisions  on  program  cost 

Quarterly 

Quarterly 

Weekly 

Understand  cost  elements  such  as  direct  labor,  direct 
materials,  general  &  administrative,  profit,  and  overhead 

Monthly 

Quarterly 

Weekly 

Understand  ways  to  characterize  costs  such  as 
fixed/variable  and  recurring/non-recutTing 

Monthly 

Monthly 

Weekly 

Be  able  to  apply  learning  curve  techniques  to  analyze 
production  costs 

Annually 
or  Less 

Annually  or 
Less 

Quarterly 

Be  able  to  select  the  appropriate  c  ^ntract  type  for  a  project 

Annually 
or  Less 

Annually  or 
Less 

Quarterly 

Understand  the  utilization  of  EAC  in  cost  management 

Understand  contractor  cost  reports  such  as  the  Cost 
Performance  Report  (CPR)  and  Cost  /  Schedule  Status 

Repmt  (CSSR) 

Monthly 

Quarterly 

Quarterly 

Be  able  to  analyze  contractor  reports  such  as  the  CPR  and 
CSSR 

Monthly 

i 

Aimually  or 
Less/ 
Quarterly 

Quarterly 

Be  able  to  evaluate  contractor-recommended  corrective 
actions  and  select  an  appropriate  course  of  action 

Monthly 

Armually  or 
Less/ 
(Quarterly 

Monthly 
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The  results  indicate  that  program  managers  in  SPOs  are  more  frequently  called 
upon  to  use  five  of  the  competencies.  The  SPOs  involvement  in  development  efforts 

could  account  for  their  increased  interest  in  economic  conditions  and  their  impact  on 

\ 

defense  contractors.  Along  the  same  lines,  SPO  efforts  in  the  early  phases  of  program 
development  are  more  likely  to  include  cost  based  contracts  which  require  analysis  of 
contractor  cost  and  schedule  status  data,  and  evaluation  of  recommended  corrective 
actions. 

The  ALCs  use  several  of  the  competencies  more  frequently  than  SPOs  or 
headquarters.  ALCs  are  likely  to  deal  with  a  large  number  of  contracts  to  procure  a 
variety  of  spare  parts  and  support  equipment  for  systems.  This  could  account  for  their 
increased  frequency  of  selecting  a  proper  contract  type,  characterizing  costs,  and 
considering  cost  elements.  Increased  consideration  of  production  rates  and  quantities 
could  be  related  to  the  ALCs  involvement  in  production  contracts.  The  longer  time 
horizon  ALCs  face  could  account  for  their  increased  consideration  of  economic  conditions 
such  as  inflation. 

The  headquarters  group  appears  to  use  many  of  the  competencies  in  Table  38  less 
frequently  than  the  other  two  groups.  Although  classified  as  program  managers, 
personnel  in  headquarters  are  not  likely  to  be  involved  in  the  day-to-day  planning  and 
management  of  programs.  Therefore,  it  is  not  surprising  that  they  do  not  frequently  use 
cost  management  competencies. 
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Primary  Program  Activity 

The  primary  program  activity  is  comparable  to  the  program  phase.  Not  all 
programs  complete  the  normal  program  phases.  A  laboratory  program,  for  example,  may 
consist  of  concept  analysis  alone.  An  ALC  program  may  focus  completely  on  a  mature 
system.  Identification  of  the  basis  of  primary  program  activity  accommodates  these  cases 
in  addition  to  conventional  phased  programs.  Nine  differences  were  identified  on  this 
basis.  Tables  41  and  42  present  the  competencies  with  response  distribution  differences. 


Table  41.  Frequency  Differences  Based  On  Primary  Activity 


Subgroup  Median  I 

Competency 

Concepts 

Technologies 

Developmental 

Systems 

Annually  or 
Less 

Annually  or  Less 

Annually  or  Less 

Quarterly 

Understand  contractor  cost  lepons 
such  as  the  Cost  Performance 

Report  (CPR)  and  Cost  /  Schedule 
Status  Report  (CSSR) 

(Quarterly 

Monthly 

Monthly 

Quarterly 

Be  able  to  analyze  contractor 
reports  such  as  the  CPR  and  CSSR 

Quarterly 

Quarterly 

Monthly 

Quarterly 

Be  able  to  evaluate  contractor- 
recommended  corrective  actions 
and  select  an  appropriate  course  of 
action 

Annually  or 
Less 

Quarterly 

Monthly 

Monthly 

Three  of  the  four  competencies  rejected  based  on  frequency  of  use  deal  with  cost 
controls  and  evaluating  contractor  performance.  The  median  values  reflect  an  increased 
frequency  of  evaluating  contractor-recommended  corrective  actions  as  the  system 
matures.  The  CPR  and  C/SSR  competencies  may  also  reflect  an  increasing  trend  with 
system  maturity  although  the  mature  system  medians  drop  back  down.  A  potential 
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contributor  to  the  decreased  frequency  of  analysis  and  understanding  of  contractor  cost 
and  schedule  reports  is  the  fact  that  firm  fixed  price  contracts,  which  are  more  common  on 
production  programs  and  commodity  purchases,  do  not  require  cost  and  schedule 
reporting. 


Table  42.  Importance  Differences  Based  on  Primary  Activity 


Subgroup  Median 

Competency 

Concepts 

Technologies 

Developmental 

Systems 

Understand  the  impact  of  production 
rate  and  quantity  decisions  on 
proKiam  cost 

Extremely 

Important 

Very  Importan.t 

Extremely 

Important 

Very 

Important 

Understand  the  impact  of  budget 
cuts  on  unit  marginal  cost 

Important 

Very  Important 

Very 

Important 

mpu 

Very  Important 

Extremely 

Important 

Very 

Important 

Understand  congressional  reporting 
requirements 

Important 

Important 

Very  Important 

Understand  the  impact  of  changes  in 
scope  on  the  cost  of  defense 
contracts 

Important 

Very  Important 

Extremely 

Important 

Very 

Important 

There  appears  to  be  an  increased  importance  placed  on  several  competencies 
during  the  developmental  systems  period.  Since  this  is  the  time  when  key  decisions 
involving  production  of  the  system  are  made,  it  seems  logical  that  consideration  of  budget 
cuts,  scope  changes,  and  production  contract  types  would  be  critical  to  the  success  of  the 
program.  A  somewhat  surprising  result  is  the  great  importance  those  focusing  on 
concepts  placed  on  consideration  of  production  rates  and  quantities.  However,  those 
doing  concept  studies  for  major  programs  may  very  well  be  called  upon  to  weigh  these 
issues  in  determining  the  budget  required  for  the  effort. 
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Position  Certification  Level 

Ten  differences  were  identified  between  the  intermediate  and  senior  level  program 
managers.  Tables  43  and  44  present  these  competencies  and  the  subgroup  median  values. 
In  comparing  subgroup  medians,  it  is  important  to  note  that  the  response  distributions  may 
be  different  even  if  the  median  values  are  identical. 


Table  43.  Frequency  Differences  Based  On  Position  Level 


Subgroup  Median  | 

Competency 

Level  n 

Level  in 

Quarterly 

Monthly 

Understand  the  products  of  cost  manaeement  software  tools 

Quarterly 

Monthly 

Understand  the  utilization  of  EAC  in  cost  manaRement 

Quarterly 

Understand  contractor  cost  reports  such  as  the  Cost  Performance  Report 
(CPR)  and  Cost  /  Schedule  Status  Report  (CSSR) 

Monthly 

Monthly 

Be  able  to  analyze  contractor  reports  such  as  the  CPR  and  CSSR 

Be  able  to  evaluate  contractor-recommended  corrective  actitms  and  select  an 
appropriate  course  of  action 

Monthly 

Monthly 

Be  able  to  develop  corrective  actions  to  counter  unfavorable  program 
variances 

Quarterly 

Monthly 

It  appears  that  the  Level  II  program  managers  are  less  frequently  called  upon  to 
use  many  of  the  competencies  in  Table  43. 


Table  44.  Importance  Differences  Based  On  Position  Level 


Subgroup  Median  | 

Competency 

Level  n 

Level  ni 

Be  able  to  apply  learning  curve  techniques  to  analyze  production  costs 

Understand  the  concept  of  earned  value  and  methods  for  calculating  it 

Important 

Understand  the  utilization  of  EAC  in  cost  management 

BrmCTiRTiiM 
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An  additional  comparison  between  intermediate  and  senior  level  personnel  was 
made  by  determining  the  number  of  application  (be  able  to)  and  comprehension 
(understand)  competencies  valuable  to  each  level.  Figure  10  illustrates  this  comparison. 


Figure  10.  Application  vs.  Comprehension  Competencies  by  Position  Level 


The  data  does  not  support  a  hypothesis  that  intermediate  level  program  managers 
would  be  required  to  both  comprehend  and  apply  cost  management  competencies,  while 
senior  level  managers  would  only  be  required  to  comprehend  the  results.  In  fact,  both 
intermediate  and  senior  level  program  managers  value  comprehension  over  application. 
Not  only  do  intermediate  and  senior  level  program  managers  value  a  similar  number  of 
comprehension  and  application  competencies,  they  value  almost  the  exact  same 
competencies.  The  only  exception  was  that  senior  level  managers  valued  two  additional 
competencies: 

•  Understand  the  impact  of  the  time  value  of  money  on  financing  and  budgeting 
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•  Understand  the  concept  of  earned  value  and  methods  for  calculating  it 
The  existence  of  program  control  and  financial  management  staffs  to  support 


program  managers  may  contribute  to  the  tendency  of  both  levels  of  program  manager  to 
favor  comprehension  over  application  by  relieving  them  of  the  necessity  to  complete  the 
analyses  themselves. 

Omitted  Competencies 

An  open-ended  question  in  the  final  section  of  the  instrument  asked  if  any 
important  cost  management  competencies  were  omitted  from  the  model.  The  responses 
varied  from  topic  areas  that  were  already  covered  in  the  original  model,  to  new  concepts 
that  had  not  been  considered.  Some  comments  addressed  topic  areas  from  the  original 
model,  but  at  a  different  level  of  detail.  Table  45  summarizes  the  topic  areas  in  the 
responses. 


Table  45.  Open-Ended  Response  Topics 


Topic  Area 

Appropriations  (type  of  funds,  year  of  funds,  expiring  funds,  unobligated 
funds,  obligation  and  expenditure  rates,  etc.) 

17 

Legal  &  regulatory  issues  impacting  cost  management  (Anti-Deficiency, 

Buy  American,  “Bonafide  Need”  rule,  etc.) 

10 

Integrated  Cost/Schedule/Requirements  Management 

6 

Evaluating  cost  proposals  for  source  selections  and  ECPs 

5 

POM  inputs  and  exercises  /  PPBS  process 

4 

Contract  termination  issues 

4 

Internal  cost  management  (supplies,  TDY,  personnel,  etc.) 

3 

Risk  evaluation  and  management 

3 

Sustainment 

3 

Intergovernmental  relationships  and  funds  transfer  (MIPRs,  MORDs) 

2 

Handling  cost  overruns 

2 
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Topic  Area 

Impact  of  overhead  costs  and  methods  for  calculating  overhead  rates 

2 

Activity  Based  Costing 

1 

Cost  management  issues  in  foreign  military  sales 

1 

Pricing 

1 

Cost  management  issues  associated  with  GFP/GFE/GFI 

1 

Contractor  cash  flow 

1 

Development  of  a  CPAR 

1 

Understand  outcomes  of  contractor  stockholder  meetings 

1 

Teaming  with  the  contractor 

1 

Logistics  cost  management 

1 

Stock  fund  issues 

1 

Industrial  base  issues  and  cost  management 

1 

Relationship  between  EAC  and  year  of  funds 

1 

Expansion  of  contract  type  selection  topic  area 

1 

1 

Analyze  contractor  work  packages 

1 

Understand  /  evaluate  contractor  resource  allocation  systems 

1 

Lack  of  an  accepted  retum-on-investment  method  or  rate 

1 

Other  government  costs 

1 

Impact  of  politics  on  cost  estimates 

1 

Cost  Management  Metrics 

1 

Negotiation 

1 

Understanding  government  systems  for  tracking  and  reporting 
commitments,  obligations,  and  expenditures 

1 

Sensitivity  analyses  on  cost  estimates 

1 

Real-time  cost  management 

1 

Understand  how  contractor  cost  account  managers  work 

1 

The  most  commonly  addressed  area  was  appropriations.  The  comments  dealt  with 
several  specific  areas  under  the  appropriations  umbrella.  There  appears  to  be  enough 
complexity  in  this  area  to  justify  expanding  its  coverage  in  the  model.  Several  respondents 
noted  important  legal  and  regulatory  issues  not  addressed  in  the  original  model  including 
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the  Anti-Deficiency  and  Buy  American  Acts.  This  area  could  be  addressed  in  a  future 
revision  of  the  cost  management  model. 

Conclusion 

The  data  provided  by  the  330  respondents  indicates  that  29  of  the  47  competencies 
in  the  model  are  considered  valuable  by  military  program  managers  in  the  field.  Many 
topics  in  the  model  consisted  of  two  competencies;  one  being  the  ability  to  complete  the 
task  and  the  other  the  ability  to  understand  the  results.  In  this  case,  26  of  the  29  valuable 
competencies  were  at  the  understanding  level,  indicating  that  program  managers  place 
little  value  on  being  able  to  complete  these  tasks  themselves.  The  existence  of  suppcnt 
staff  such  as  cost  analysts  and  financial  managers  may  contribute  to  this  perception. 

The  number  of  response  distribution  differences  identified  in  each  biographic 
category  varied.  There  was  almost  no  variation  on  the  basis  of  management  education. 
Roughly  10%  of  the  Kruskal-Wallis  tests  identified  variations  in  the  perceived  importance 
or  frequency  of  use  of  competencies  on  the  basis  of  position  certification  level  or  primary 
program  activity.  In  terms  of  organization  type,  there  were  no  differences  in  the  perceived 
importance  of  the  competencies.  However,  12  competencies  showed  differences  in 
frequency  of  use.  The  respondents  also  produced  a  number  of  topic  areas  that  should  be 
considered  for  inclusion  in  future  cost  management  competency  models. 
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V.  Conclusions  and  Recommendations 


Introduction 

Acquisition  organizations  have  historically  handled  over  35  %  of  the  DoD  budget 
(Cheney,  1993:143).  The  sheer  magnitude  of  money  involved  and  the  intense  negative 
publicity  resulting  from  failures  in  this  area  make  cost  management  critically  important  to 
DoD  program  managers  and  senior  staff.  This  research  effort  was  conducted  to  evaluate 
the  cost  management  competencies  valued  by  Air  Force  program  managers.  The  effort 
focused  on  two  major  research  questions:  What  cost  management  competencies  do 
program  managers  value,  and  how  do  these  perceptions  vary  among  subgroups  of  the 
population? 

In  order  to  answer  these  questions,  a  47  element  cost  management  competency 
model  was  developed  from  a  foundation  of  management  research  and  education  programs. 
The  key  resources  were  the  Project  Management  Institute’s  Project  Management  Body  of 
Knowledge  and  a  program  management  competency  list  developed  by  the  Acquisition 
Management  Functional  Board.  An  important  feature  of  the  model  was  the  consideration 
of  the  difference  between  the  ability  to  complete  a  task  and  the  ability  to  understand  the 
results  of  an  effort. 

The  perceptions  of  Air  Force  program  managers  were  gathered  through  a  mail 
survey  regarding  the  perceived  importance  and  frequency  of  use  of  the  competencies  in 
the  model.  The  survey  was  distributed  to  682  intermediate  and  senior  level  program 
managers  in  Air  Force  Materiel  Command.  A  total  of  330  managers  responded.  The 
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remainder  of  this  chapter  summarizes  the  study  results  and  presents  conclusions, 
recommendations,  and  suggestions  for  further  study. 

Results 

A  combined  measure  of  frequency  and  importance  indicated  that  29  of  the  47 
competencies  were  valuable  to  program  managers.  Of  these,  26  involved  understanding, 
while  only  three  involved  having  the  ability  to  actually  complete  a  task.  The  competency 
model  contained  three  major  areas:  cost  estimating  and  forecasting,  cost  budgeting,  and 
cost  control.  The  cost  control  section  showed  the  highest  proportion  of  valuable 
competencies,  followed  by  the  cost  budgeting  and  cost  estimating  and  forecasting 
sections. 

Variations  in  responses  were  examined  based  on  four  factors:  organization  type, 
management  education,  primary  program  activity,  and  the  certification  level  of  the 
position.  The  number  of  response  distribution  differences  identified  in  each  biographic 
category  varied.  There  was  almost  no  variation  on  the  basis  of  management  education. 
There  was  variance  in  the  fiequency  of  use  of  one  third  of  the  competencies  on  the  basis 
of  organization  type,  but  almost  no  variation  in  perceived  importance  on  this  basis.  There 
was  variation  in  the  perceived  importance  and  frequency  of  use  of  roughly  10  %  of  the 
competencies  on  the  basis  of  primary  program  activity  and  position  certification  level. 
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Conclusions 


The  results  of  this  research  have  direct  implications  for  defense  program  managers. 
In  particular,  the  results  are  relevant  to  the  education  and  training  of  program  managers, 
their  classification  in  the  acquisition  workforce,  and  their  professional  certification. 

Education  and  Training 

Education  and  training  programs  consume  time  and  money,  both  of  which  are 

limited  resources.  The  goal  of  these  programs  should  be  to  develop  the  competencies  that 

will  provide  the  most  benefit  to  program  managers  on  the  job.  Once  the  most  important 

competencies  have  been  identified,  the  best  method  to  convey  the  information  must  be 

determined.  Determining  the  optimum  training  method  for  the  important  competencies 

identified  in  this  study  would  require  further  research  as  discussed  below.  In  the 

meantime,  there  are  opportunities  to  improve  our  current  programs. 

The  results  of  this  study  indicate  that  program  managers  value  comprehension 

level  competencies  over  application  level  competencies.  The  existence  of  program  control 

and  financial  management  staffs  could  account  for  this  response.  The  support  staff 

completes  the  analysis  and  the  program  manager  makes  decisions  based  upon  their  results. 

This  recurrent  theme  was  reflected  in  the  comments  of  respondents: 

A  program  manager  shouldn’t  be  doing  these  detail[ed]  cost  things,  he 
should  have  experts  doing  them  and  advising  him  on  what’s  happening. 

He  should  be  schooled  in  these  things  so  he  understands  what  he  is  being 
told.  If  he  is  doing  these  detailed]  things  he  is  micro  managing  and  not 
doing  his  primary  job  of  managing  his  program  to  success! 

I’m  a  colonel  &  while  I  don’t  personally  do  a  lot  of  these  things,  folks  that 
work  for  me  do! 
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Within  an  IPT  structure,  I  rely  on  the  financial  management  members  of  the 
team  to  actually  do  the  analysis.  It  is  important  for  me  to  understand  their 
analysis  and  its  implications,  but  not  important  for  me  to  be  able  to  do  it 
myself.  I  count  on  them  to  inform  me  of  adverse  trends. 

The  prevalence  of  understanding  over  doing  was  visible  in  the  most  frequently 

used,  most  important,  and  most  valuable  competencies.  The  trend  carried  through  both 

intermediate  and  senior  level  program  managers.  The  implication  is  that  education  and 

training  programs  should  emphasize  understanding.  Instead  of  focusing  on  repeated 

execution  of  a  given  competency,  programs  might  discuss  and  demonstrate  the 

competency,  but  focus  on  understanding  the  implications  of  the  results.  The  point  is  not 

that  doing  the  task  will  not  enhance  understanding.  Rather,  it  is  a  question  of  emphasis, 

placing  less  emphasis  on  the  technique  itself  than  on  how  the  results  might  impact  the 

actions  of  a  program  manager. 

For  example,  the  methods  for  calculating  estimates  at  completion  can  be  taught 
and  demonstrated.  The  instruction  ought  then  be  extended  to  include  a  discussion  of  the 
management  decisions  that  the  results  might  dictate.  This  approach  expands  the  narrow 
topic  to  the  bigger  picture.  An  alternate  approach  is  to  incorporate  some  details  in 
broader  discussions.  This  points  toward  a  case  study  or  situational  analysis  approach.  In 
this  case,  the  discussion  of  a  case  might  present  actual  estimate  at  completion  values.  The 
decisions  made  by  the  program  managers  and  their  impact  on  the  program  can  then  be 
analyzed. 

Respondents  mentioned  the  need  to  complete  what  if  exercises  with  budgets  and 
cost  estimates.  The  results  also  showed  that  competencies  dealing  with  the  impact  of 
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changes  in  scope  and  corrective  actions  were  valuable  to  program  managers.  The 
common  element  was  assessing  the  impact  of  changes  in  inputs,  resources,  rates, 
quantities,  schedule,  etc.  This  concept  can  be  incorporated  into  education  and  training 
programs  through  question  structure.  Instead  of  providing  a  single,  work-through 
question  using  the  competency,  follow  the  question  up  with  some  change  questions.  What 
is  the  impact  of  cutting  delivery  quantity  in  half  on  unit  price?  What  is  the  schedule  impact 
of  a  30%  cut  in  manpower?  Which  production  method  is  optimum  if  the  price  of  materials 
increases  by  20%  ? 

The  results  also  point  out  some  areas  that  may  receive  relatively  less  emphasis  in 
many  education  and  trairung  programs.  The  competency  most  frequently  used  by  the 
program  managers  surveyed  was  understanding  the  impact  of  the  political  process  on 
acqmsition  management.  The  results  also  highlighted  the  value  of  understanding  the 
PPBS  and  congressional  budgeting  and  appropriations  processes.  These  processes  are 
clearly  subject  to  political  influence,  which  only  serves  to  enhance  the  value  of  improving 
the  competency  of  program  managers  in  operating  in  the  political  environment.  The 
tenuous  nature  of  operating  in  the  political  environment  makes  it  difficult  to  convey  in  a 
conventional,  textbook  approach,  and  this  may  result  in  a  decreased  emphasis  on  this  area 
in  training.  However,  the  importance  of  competence  in  this  area  is  clear,  and  the  subject 
may  well  be  suited  to  the  case  study  or  situational  analysis  methods  of  instruction. 
Thamhain  shows  that  experiental  learning  and  observation  account  for  84%  of  project 
managers  professional  development  (1991 : 41).  Developing  competence  operating  in  the 
political  environment  is  probably  particularly  suited  to  these  two  methods. 
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Classification  of  Program  Managers 

Under  APDP  and  DODD  5000.2,  program  managers  are  a  distinct  group  of 

professionals  in  the  acquisition  workforce,  but  only  one  of  many.  The  most  common  place 

people  envision  program  managers  is  in  a  program  office,  and  the  data  reflected  this. 

However,  there  are  a  number  of  program  management  positions  outside  of  the  SPOs.  One 

person  who  declined  to  complete  the  survey  wrote: 

I  am  unable  to  respond  to  your  questionaire  [sicj.  The  duties  of  this  office 
involve  the  day  to  day  administration  of  the  product  center.  This  office  is 
not  doing  product/program  management.  Although  I  have  a  program 
management  background,  my  current  position  tasks  do  not  lend  themselves 
to  what  is  asked  in  your  questionaire  [sic]. 

Other  similar  responses  raise  the  question  of  whether  the  variety  of  jobs  currently 
covered  under  the  program  management  umbrella  are  similar  enough  to  be  included  in  a 
single  group.  For  the  most  pan,  this  study  supports  the  conclusion  that  they  are,  in  terms 
of  cost  management.  One  could  theorize  that  the  cost  management  activities  required  of 
different  organization  types  and  program  activities  would  vary,  but  there  were  few 
significant  differences  in  the  responses  from  these  groups.  The  largest  discrepancy  was 
the  difference  in  the  frequency  of  use  of  15  competencies  among  the  various  organization 
types.  However,  the  program  office  was  clearly  the  dominant  organization  type  and  in  the 
coming  era  of  cradle-to-grave  management  in  the  SPO,  the  distinction  between  some  of 
the  organization  types  may  blur.  Looking  at  the  big  picture,  the  differences  identified  in 
this  study  do  not  justify  the  additional  administrative  burden  of  managing  program 
managers  at  a  lower  level. 


92 


Certification  Programs 

Program  managers  are  currently  certified  under  two  sets  of  criteria.  Under  APDP, 
they  are  certified  at  one  of  three  levels  according  to  their  experience,  education,  and 
training.  Program  management  positions  are  also  coded  with  the  certification  level  desired 
of  the  person  filling  that  job.  A  similar  set  of  criteria  is  used  to  identify  program  managers 
as  eligible  for  the  acquisition  corps.  Those  actually  serving  in  positions  identified  as 
critical  acquisition  positions  are  considered  members  of  the  corps. 

The  presumption  is  that  those  meeting  the  criteria  for  certification  are  qualified 
for  all  positions  at  that  level.  Unfortunately,  the  criteria  are  quite  general.  No  specific 
criteria  exist  for  individual  skill  areas  such  as  cost  management.  Even  if  validated 
competency  models  existed,  the  administrative  burden  of  tracking  to  this  level  of  detail 
and  the  difficulty  of  evaluating  competence  preclude  the  use  of  more  specific  criteria. 
Despite  the  desire  to  develop  a  certification  system  identifying  program  managers  as 
qualified  for  a  broad  variety  of  jobs,  it  appears  that  individual  qualifications  and  experience 
will,  and  should,  continue  to  play  a  substantial  role  in  the  selection  of  officers  for  specific 
positions.  Input  from  the  gaining  unit  should  thus  be  an  important  contribution  to  the 
Military  Personnel  Center’s  selection  decision. 

Recommendations 

The  results  of  this  study  have  direct  implications  for  the  education  and  training  of 
defense  program  managers.  In  order  to  maximize  the  benefit  received  for  the  time  and 
money  expended  in  education  and  training  programs,  these  programs  should  be  focused  on 
the  competencies  that  will  prove  most  valuable  to  program  managers  in  their  jobs.  The  29 
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valuable  cranpetencies  identified  in  this  effort  can  serve  as  a  foundation  for  these 
decisions.  The  list  might  be  expanded  to  account  for  the  subjective  nature  of  the  decision 
rule  used  to  identify  the  valuable  competencies.  Respondents  identified  several  additional 
areas  not  included  in  the  original  model  that  could  be  evaluated  through  further  research. 

The  results  also  indicate  that  program  managers  value  understanding  over  the 
ability  to  complete  a  particular  task.  Education  and  training  programs  should  therefore 
emphasize  understanding  the  implications  of  cost  management  activities  over  rote 
memorization  and  repeated  execution  of  a  particular  task.  This  can  be  accomplished  by 
expanding  the  discussion  of  the  topic  after  teaching  and  demonstrating  a  technique.  The 
case  study  method  is  another  excellent  method  for  addressing  the  impact  of  cost  analysis 
on  program  management  decisions. 

The  existence  of  cost  analysts  and  financial  managers  to  suppon  the  program 
manager  appeared  to  play  a  critical  role  in  shaping  the  perceptions  of  program  managers. 
The  ability  of  support  staffs  to  complete  analyses  leaves  the  program  manager  free  to 
consider  the  implications  of  the  results.  Further  research  is  warranted  to  clarify  the 
relationships  between  these  support  personnel  and  the  program  manager. 

Another  important  program  management  task  is  assessing  the  impact  of  program 
changes.  Program  managers  are  frequently  called  upon  to  assess  the  impact  of  changes  in 
the  funding,  schedule,  and  scope  of  their  programs.  Education  and  training  programs 
could  incorporate  consideration  of  changes  into  their  question  structure.  Instead  of 
requiring  only  a  rote  solution  to  a  fixed  problem,  include  additional  questions  regarding 
the  impact  of  changing  the  inputs  to  the  process  or  the  desired  output. 
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This  results  of  this  study  do  not  appear  to  point  toward  any  major  changes  in 
classification  and  certification  programs.  The  number  of  differences  in  perceptions 
identified  within  the  program  management  field  do  not  appear  to  justify  more  detailed 
classification.  While  this  study  could  lead  to  improved  education  and  training  programs,  it 
does  not  indicate  that  changes  are  necessary  in  certification  programs  such  as  APDP.  The 
administrative  burden  of  tracking  the  competence  of  program  managers  in  detailed  cost 
management  areas  does  not  appear  justified  givei*  the  difficulty  and  subjectivity  involved  in 
evaluating  competence. 

Follow-On  Research 

There  are  many  opportunities  for  continuing  this  research.  These  opportunities 
vary  in  scope  and  potential  significance.  The  opportunities  include  modification  and  re- 
evaluation  of  the  model,  expanding  the  research  to  a  broader  population,  extending  the 
research  to  other  functional  areas,  and  increasing  the  depth  of  the  research  by  considering 
level  of  competency  and  training. 

Modifying  the  Model 

The  open-ended  responses  of  the  program  managers  provided  valuable  input  on 
the  scope  and  c  ntent  of  the  competency  model.  The  difficulty  is  developing  a  model  that 
covers  the  important  aspects  of  cost  management  without  being  so  cumbersome  that  it  is 
difficult  to  validate  and  difficult  to  use  as  a  tool  for  structuring  education  and  training 
programs.  One  possible  research  effort  would  integrate  comments  from  the  respondents 
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in  this  effort  into  a  revised  model.  The  revised  model  could  then  be  evaluaeed  by  a  similar 
group  of  program  managers. 

Determining  the  appropriate  level  of  indenture  for  each  competency  is  a  problem  in 
developing  models  of  this  type.  Less  important  topics  can  be  covered  at  a  high  level  while 
more  important  and  more  complex  topics  should  be  covered  in  more  detail.  This  problem 
was  identified  in  at  least  one  area  of  this  study.  A  number  of  comments  dealt  with 
portions  of  the  appropriations  process.  In  particular,  managers  noted  the  importance  of 
items  such  as  obligation  and  expenditure  rates,  unobligated  funds,  expiring  funds,  type  of 
money  and  year  of  money.  The  competency  in  the  model  dealing  directly  with  the 
congressional  budgeting  and  appropriations  process  could  be  expanded  to  reflect  some  of 
the  particularly  important  elements  individually. 

There  were  several  comments  regarding  the  importance  of  integrating  the 
management  of  cost,  schedule,  and  requirements.  This  area  includes  such  concepts  as  an 
integrated  master  schedule,  design-to-cost,  and  relating  technical  and  schedule 
requirements  to  dollar  amounts.  Although  these  concepts  are  fairly  high  level  in  the  cost 
management  functional  area,  they  are  probably  worthy  of  mention  in  at  least  a  single, 
general  competency. 

Although  the  model  contained  a  competency  dealing  with  understanding  the 
organization’s  role  in  the  PPBS  process,  several  respondents  commented  on  the  POM 
process.  The  competency  statement  could  be  rewritten  to  more  clearly  reflect  that  the 
POM  inputs  are  an  organization’s  major  contribution  to  the  PPBS  process. 
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Several  topic  areas  that  were  not  included  in  the  original  model  may  be  worthy  of 

inclusion  in  a  cost  management  model.  The  importance  of  legal  and  regulatory  issues  such 

as  anti-deficiency  were  noted  in  several  responses.  The  original  model  did  not  discuss 

relationships  and  transfers  of  funds  between  government  organizations.  Other  significant 

areas  included  activity  based  costing,  contract  termination  issues,  foreign  military  sales, 

government  furnished  property  or  information,  and  pricing  issues. 

Another  issue  is  the  management  of  costs  within  the  organization  such  as  travel, 

supplies,  and  support  personnel.  As  one  respondent  noted: 

what  labs  need  (and  other  organization[s])  are  the  tools  and  capabilities  to 
do  cost  accounting  &  management  the  same  way  we  demand  of 
contractors.  How  much  does  it  cost  to  build  something  in  a  Lab,  where  do 
the  dollars  go,  how  are  the  schedules  measured  &  maintained?  There  is  no 
capability  or  commitment  to  execute  in  this  manner  but  with  falling 
budgets,  we  must  get  more  efficient 

This  area  is  probably  neglected  in  most  education  and  training  program  in  favor  of 
emphasis  on  SPO  requirements.  This  area  is  probably  worthy  of  inclusion  in  a  model. 

Broader  Population 

This  research  focused  on  military  program  managers  within  Air  Force  Materiel 
Command.  The  basic  structure  of  this  effort  could  be  duplicated  with  a  broader 
population  of  program  managers  including  civilian  program  managers  or  other  military 
services.  Although  civilian  and  military  program  managers  work  together  on  a  daily  basis, 
they  often  have  different  backgrounds  and  job  experience.  Military  program  managers  are 
likely  to  have  a  broader  variety  of  experience,  while  civilians  may  have  more  depth.  It  is 
therefore  possible  that  military  and  civilian  program  managers  value  different  cost 
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manageinent  competencies.  Identifying  differences  between  the  two  groups  could 
highlight  strengths  and  weaknesses  resulting  from  their  backgrounds  and  training.  There 
is  potential  for  improved  education  and  training  that  more  specifically  targets  the  needs  of 
each  group. 

The  same  argument  supports  studying  other  military  services.  Despite  tight 
budgets  and  an  emphasis  on  joint  programs,  each  service  maintains  its  own  system  for 
managing  program  costs  and  for  educating  and  training  its  program  managers.  Programs 
such  as  DSMC’s  Program  Management  Course  may  provide  some  commonality,  but 
program  managers  are  a  product  of  their  upbringing.  Having  a  common  understanding  of 
which  cost  management  competencies  are  valuable  to  program  managers  would  allow  all 
the  military  services  to  focus  their  education  and  training  programs  on  these  areas.  This 
more  efficient  approach  to  training  could  improve  the  performance  of  program  managers 
on  both  single  service  and  joint  programs. 

Cost  Analysts  and  Financial  Managers 

The  results  of  this  effort  reflect  that  program  managers  value  comprehension  of  the 
competencies  over  their  application.  Many  respondents  cited  the  fact  that  the  analyses  and 
estimates  in  question  are  completed  by  support  staffs  and  that  program  managers  must 
simply  understand  the  question.  A  potential  research  effort  would  be  to  evaluate  the 
perceived  importance  and  frequency  of  use  of  the  competencies  by  the  cost  analysts  and 
financial  managers. 

This  effort  could  be  based  on  the  model  and  instruments  developed  in  this  study 
although  the  modifications  described  above  could  be  incorporated.  The  proposed  effort 
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would  identify  whether  me  cost  analysts  and  financial  managers  complete  the  analyses 
attributed  to  them  by  me  program  manager  or  if  the  results  come  from  contractor  reports. 
The  effort  could  also  identify  differences  in  me  perceived  value  of  various  competencies 
among  program  managers,  cost  analysts,  and  frnancial  managers.  The  study  could  also 
examine  differences  based  on  me  structure  of  me  organization.  The  roles  of  cost  analysts, 
financial  managers,  and  program  managers  may  vary  depending  upon  whemer  the 
structure  is  purely  functional,  matrix,  or  integrated  product  teams.  The  results  could 
highlight  limitations  in  me  education  and  training  of  program  managers  and  could  result  in 
better  coordination  between  mese  functional  support  areas  and  the  program  manager. 

Other  Skill  Areas 

The  Project  Management  Institute’s  Project  Management  Body  of  Knowledge  and 
me  Defense  Body  of  Knowledge  derived  by  Best  and  Kobylarz  divide  program 
management  into  a  number  of  skill  areas  of  which  cost  management  is  only  one.  A  similar 
research  effort,  involving  development  and  evaluadon  of  a  competency  model,  could  be 
conducted  in  a  number  of  other  skill  areas.  Skill  areas  mat  appear  suited  to  study  include 
quality  management,  logistics  management,  engineering  management,  manufacturing 
management,  schedule  management,  and  software  management  to  name  a  few. 

Competency  and  Training 

This  study  evaluated  the  perceived  importance  and  frequency  of  use  of  a  set  of 
cost  management  competencies  in  order  to  identify  mose  valuable  to  program  managers. 

A  significant  area  not  addressed  by  this  study  is  me  actual  level  of  competency  possessed 
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by  program  managers  in  each  area.  Program  managers  may  not  value  a  given  competency 
because  they  do  not  understand  its  application  or  benefits.  Thus,  while  education  and 
training  programs  might  be  improved  based  on  this  research,  the  picture  is  incomplete 
without  further  study. 

A  related  question  is  the  training  method.  There  are  a  number  of  questions  in  this 
area.  The  first  is:  Which  of  the  wide  variety  of  available  sources,  from  on-the-job  training 
to  formal  education,  did  program  managers  use  to  develop  the  competency?.  Although 
the  manager  may  have  learned  it  through  one  method,  it  is  also  important  to  ask  how  they 
feel  the  competency  should  be  acquired.  There  may  be  a  better  way  to  get  the  job  done. 

Addressing  these  issues  for  the  complete  competency  model  would  require  a 
significant  amount  of  time  fiom  each  respondent  An  alternative  approach  is  to  select  a 
subset  of  the  competencies  for  analysis.  Selection  could  be  based  on  criteria  such  as  the 
most  important  or  frequently  used  competencies,  the  most  valuable  competencies  (by  the 
combined  measure),  or  through  a  sample  from  each  major  category  of  the  model  (cost 
estimating  and  forecasting,  cost  budgeting,  and  cost  control). 

A  final  question  involves  identifying  when  specific  competencies  are  required  in  a 
program  manager’s  career.  The  tasks  and  knowledge  required  of  program  managers  can 
vary  as  their  careers  progress.  These  questions  could  be  answered  by  surveying  program 
managers  with  significant  experience.  The  perceptions  of  program  managers  in  the  field 
would  form  a  basis  for  identifying  the  valuable  competencies,  the  appropriate  career  point 
for  development,  and  the  optimum  training  method.  A  potential  limitation  of  this 
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approach  is  that  the  perceptions  of  current  program  managers  are  a  product  of  their 
training  and  experience,  so  they  may  be  biased  toward  elements  of  the  current  system. 
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Appendix  A:  Survey  Instruments 


USAf  SCN;  94-15 


memorandl'm  for  survey  respondents 

FROM:  AFPEO/CM 

1060  Air  Fores  Penagon 
Washington  DC  20330-1060 

SUBJECT;  Program  Managsr  Cost  Manageasnt  Survey  Package  -  ACTION  MEMOR.ANDUM 

Effective  cost  management  is  ciucai  a  the  success  of  OoO  acquisition  programs.  It  is 
thereeore  impottant  to  determine  which  cost  management  practices  are  United  to  ei^ecdve 
program  man^ment.  You  were  selected  to  paidcipaie  in  this  teseareh  based  on  your 
expetieoce.  educadon.  and  training  in  the  area  of  DoO  program  management  Your  pardcipaiion 
will  greatly  facilitate  our  efforts  to  influence  Ae  saining  and  edncadon  of  program  managers. 

This  is  not  a  test  and  there  are  no  tight  or  wrong  answers.  It  is  impottant  that  you 
responds)  each  statement  as  thoughtfully  and  dankiy  as  possible.  Pmtesi  of  this  survey  indicates 
that  it  will  take  you  approximaieiy  1 1  minunw  to  complete  it 

Participation  in  this  leseatch  is  voluntary,  but  your  conoibuiion  is  impottant  and  will  be 
used  as  an  input  to  improve  training  and  education  programs.  No  effort  will  be  made  to  assodate 
you  with  your  response.  Ndnpatdcipation  will  not  result  in  any  adverse  action.  Please  remnx  this 
survey  package  no  later  than  1  May  1994  to  the  address  iadfcaied  on  the  survey  package.  For 
farther  infonaadon.  contaa  Capt  Brent  Batter  at  AFTT/LA.  DSN  785-77T7,  e.xt  2109. 


TEDDY  L.  HOUSTON 
Associate  Diiecntr  of  Acquisidon 
Career  Management 
Assisont  Secieaiy  (Acquisidon) 


3  Attachments: 

1.  Survey 
1  AFirfom  lie 
3.  Retum  Envelope 
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Program  Manager  Cost  Management  Survey 


Instructions: 

•  Use  a  number  2  pencil 

•  Please  read  each  question  and  darken  the  appropriate  circle  on  the  answer 
sheet  provided 

•  Space  for  written  comments  has  been  provided  at  the  end  of  the  survey 

•  Your  responses  will  remain  anonymous.  Please  do  not  put  your  name  on  the 
answer  sheet 

•  Thank  you  for  your  time 

•  If  you  have  any  questions  please  contact  Capt  Brent  Baxter  at: 

DSN:  785-7777x2109 
Commercial:  (513)255-7777  x2109 
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Section  1;  Biographical  Questions 

1.  Which  of  the  following  advanced  management  education  programs  have  you 
completed? 

A.  None 

B.  Graduate  Management  Degree 

C.  Defense  Systems  Management  College  (Program  Management  Course) 

D.  Both  B  and  C 

2.  In  what  type  of  organization  do  you  work? 

A.  System  Program  Office 

B.  Laboratory 

C.  Headquaners 

D.  Air  Logistics  Center 

E.  Other _ . 

3.  With  which  do  you  generally  work? 

A.  Concepts  (Basic  Research,  Concept  Exploration) 

B.  Technologies  (Technology  Development,  DemonstrationA^alidation) 

C.  Developmental  Systems  (Engineering  and  Manufacturing  Development) 

D.  Mature  Systems  (Production,  Deployment,  and  Support) 


Please  Turn  to  the  Next  Page 
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Section!:  Importance 

In  this  section  you  will  evaluate  program  manager  competencies  in  terms  of  importance. 
Please  note  the  response  scale  when  answering  the  questions. 


Importance 


1 

2 

3 

4 

5 

6 

Not 

Important 

Slightly 

Important 

Important 

Very 

Important 

Don’t  Know 

How  important  is  it  for  you  to: 

4.  Understand  current  economic  conditions  and  their  impact  on  defense  contractors 

5.  Be  able  to  evaluate  contractor  financial  health  and  viability  using  financial  statements 

6.  Understand  the  results  of  financial  statement  analysis  conducted  to  evaluate  contractor 
financial  health 

7.  Be  able  to  apply  time  value  of  money  techniques  such  as  return  on  investment  (ROl), 
net  present  value  (NPV),  internal  rate  of  return  (IRR),  and  Discounted  Cash  Flow  (DCF) 

8.  Understand  the  impact  of  the  time  value  of  money  on  financing  and  budgeting 

9.  Understand  how  contractors  secure  funds  to  support  ongoing  projects,  plant 
improvements,  and  new  product  development 

10.  Be  able  to  develop  a  cost  estimate  using  appropriate  methods  (e.g.  parametric, 
analogy,  grass  roots) 

1 1.  Understand  cost  estimates  developed  using  appropriate  methods  (e.g.  parametric, 
analogy,  grass  roots) 

12.  Be  able  to  use  statistical  analysis  methods  such  as  range  analysis  and  confidence 
intervals  to  characterize  the  uncertainty  associated  with  cost  estimates 

13.  Understand  the  implications  of  uncertainty  associated  with  cost  estimates 

14.  Be  able  to  u.se  software  tools  to  suppon  cost  estimation,  cost  analysis,  and 
presentation 

15.  Understand  the  products  of  cost  management  software  tools 

16.  Understand  how  contractors  apply  management  reserve  to  respond  to  contingencies 
over  the  duration  of  a  contract 

17.  Be  able  to  apply  inflation  factors  to  program  costs  and  funding 

18.  Understand  the  impact  of  inflation  on  program  costs  and  funding 

19.  Understand  the  impact  of  production  rate  and  quantity  decisions  on  program  cost 

20.  Understand  the  impact  of  budget  cuts  on  unit  marginal  cost 

21.  Understand  cost  elements  such  as  direct  labor,  direct  materids,  general  & 
administrative,  profit,  and  overhead 

22.  Understand  ways  to  characterize  costs  such  as  fixedA^ariable  and  recurring/non- 
rectirring 

23.  Be  able  to  do  break-even  analysis  using  the  concepts  of  fixed  and  variable  costs 

24.  Understand  the  cost  concepts  of  reasonableness,  allocability,  and  allowability 

25.  Be  able  to  apply  learning  curve  techniques  to  analyze  production  costs 
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Importance 


1 _ ! _ 

2 

3 

4 

5 

6 

Slightly 

Important 

Important 

Very 

Important 

Don’t  Know 

How  important  is  it  for  you  to: 

26.  Understand  the  impact  of  learning  curves  on  production  costs 

27.  Understand  the  impact  of  the  time  value  of  money  on  financing  and  budgeting 

28.  Be  able  to  develop  a  Work  Breakdown  Structure  (WBS)  that  describes  the  entire 
work  effort 

29.  Understand  the  uses  of  a  WBS  for  cost  management 

30.  Understand  the  role  the  organization  plays  in  the  Planning,  Programming,  and 
Budgeting  System  (PPBS) 

3 1 .  Understand  the  congressional  budgeting  and  appropriations  processes 

32.  Understand  the  flow  of  funds  through  the  expenditure  categories  of  commitments, 
obligations,  and  expenditures 

33.  Be  able  to  distribute  the  cost  of  work  packages  across  the  time  horizon  to  develop  a 
performance  measurement  baseline  (PMB) 

34.  Understand  the  composition  of  a  PMB 

35.  Be  able  to  evaluate  contractor  cost  accounting  and  control  systems 

36.  Understand  contractor  cost  accounting  aiid  control  systems 

37.  Be  able  to  select  the  appropriate  contract  type  for  a  project 

38.  Understand  the  cost  implications  of  alternative  contract  types  and  pricing  mechanisms 

39.  Understand  the  impact  of  the  political  environment  on  acquisition  management 

40.  Be  able  to  generate  congressionally  required  repons  such  as  the  Selected  Acquisition 
Report  (SAR) 

41.  U nder stand  congressional  reporting  requirements 

42.  Understand  the  legal  and  regulatory  requirements  for  cost  and  schedule  control 
systems  such  as  C/SCSC 

43.  Understand  the  impact  of  changes  in  scope  on  the  cost  of  defense  contracts 

44.  Be  able  to  estimate  earned  value  using  methods  such  as  weighted  milestones  and 
percent  complete 

45.  Understand  the  concept  of  earned  value  and  methods  for  calculating  it 

46.  Be  able  to  develop  an  estimate  at  completion  (EAC)  based  on  the  data  presented  in 
contractor  performance  reports 

47.  Understand  the  use  of  estimates  at  completion  (EAC)  in  cost  management 

48.  Be  able  to  analyze  contractor  reports  such  as  the  CPR  and  CSSR 

49.  Understand  contractor  cost  reports  such  as  the  Cost  Performance  Report  (CPR)  and 
Cost  /  Schedule  Status  Report  (CSSR) 

50.  Be  able  to  evaluate  contractor-recommended  corrective  actions  and  select  an 
appropriate  course  of  action 

51.  Be  able  to  develop  corrective  actions  to  counter  unfavorable  program  variances 
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Section  3:  Frequency 

In  this  section  you  will  evaluate  program  manager  competencies  in  terms  of  frequency  of 
use.  Please  note  the  new  response  scale. 


Frequency 


r  ~ — 

2 

3 

4 

5 

6 

Don’t  Know 

How  frequently  are  you  called  upon  to: 

52.  Apply  time  value  of  money  techniques  such  as  return  on  investment  (ROI),  net 
present  value  (NPV),  internal  rate  of  return  (IRR),  and  Declining  Cash  Flow  (DCF) 

53.  Understand  the  impact  of  the  time  value  of  money  on  financing  and  budgeting 

54.  Understand  how  contractors  secure  funds  to  support  ongoing  projects,  plant 
improvements,  and  new  product  development 

55.  Develop  a  cost  estimate  using  appropriate  methods  (e.g.  parametric,  analogy,  grass 
roots) 

56.  Understand  cost  estimates  developed  using  appropriate  methods  (e.g.  parametric, 
analogy,  grass  roots) 
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Comments 

•  Do  you  feel  that  any  significant  program  manager  cost  management  competencies 
were  omitted? 


•  What  single  activity  related  to  cost  management  is  the  most  important  to  you  and  what 
activity  is  most  frequendy  used? 


•  Any  other  comments? 


m 


Program  Manager  Cost  Management  Survey 


Instructions: 

•  Use  a  number  2  pencil 

•  Please  read  each  question  and  darken  die  appropriate  circle  on  the  answer 
sheet  provided 

•  Space  for  written  comments  has  been  provided  at  the  end  of  the  survey 

•  Your  responses  will  remain  anonymous.  Please  do  not  put  your  name  on  the 
answer  sheet 

•  Thank  you  for  your  time 

•  If  you  have  any  questions  please  contact  Capt  Brent  Baxter  at: 

DSN:  785-7777x2109 
Commercial:  (513)  255-7777  x2109 
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Section  1;  Biographical  Questions 

1.  Which  of  the  following  advanced  management  education  programs  have  you 
completed? 

A.  None 

B.  Graduate  Management  Degree 

C.  Defense  Systems  Management  College  (Program  Management  Course) 

D.  Both  B  and  C 

2.  In  what  type  of  organization  do  you  work? 

A.  System  Program  Office 

B.  Laboratory 

C.  Headquarters 

D.  Air  Logistics  Center 

E.  Other _ . 

3.  With  which  do  you  generally  work? 

A.  Concepts  (Basic  Research,  Concept  Exploration) 

B.  Technologies  (Technology  Development,  Concept  Demonstration) 

C.  Developmental  Systems  (Engineering  and  Manufacturing  Development) 

D.  Mature  Systems  (Production,  Deployment,  and  Support) 


Please  Turn  to  the  Next  Page 
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Section  2;  Frequency 

In  this  section  you  will  evaluate  program  manager  competencies  in  terms  of  frequency  of 
use.  Please  note  the  new  response  scale. 


Frequency 


1 _ 1 _ 

2 

3 

4 

5 

6 

Monthly 

Daily 

Don’t  Know 

How  frequently  are  you  called  upon  to: 

4.  Understand  current  economic  conditions  and  dieir  impact  on  defense  contractors 

5.  Evaluate  contractor  financial  health  and  viability  using  financial  statements 

6.  Understand  the  results  of  financial  statement  analysis  conducted  to  evaluate  contractor 
financial  health 

7.  Apply  time  value  of  money  techniques  such  as  return  on  investment  (ROI),  net  present 
value  (NPV),  internal  rate  of  return  (IRR),  and  Declining  Cash  Flow  (DCF) 

8.  Understand  the  impact  of  the  time  value  of  money  on  financing  and  budgeting 

9.  Understand  how  contractors  secure  funds  to  support  ongoing  projects,  plant 
improvements,  and  new  product  development 

10.  Develop  a  cost  estimate  using  appropriate  methods  (e.g.  parametric,  analogy,  grass 
roots) 

1 1 .  Analyze  contractor  reports  such  as  the  CPR  and  CSSR 

12.  Understand  cost  estimates  developed  using  appropriate  methods  (e.g.  parametric, 
analogy,  grass  roots) 

13.  Use  statistical  analysis  methods  such  as  range  analysis  and  confidence  intervals  to 
characterize  the  uncertainty  associated  with  cost  estimates 

14.  Understand  the  implications  of  uncertainty  associated  with  cost  estimates 

15.  Use  software  tools  to  support  cost  estimation,  cost  analysis,  and  presentation 

16.  Understand  the  products  of  cost  management  software  tools 

17.  Understand  how  contractors  apply  management  reserve  to  respond  to  contingencies 
over  the  duration  of  a  contract 

18.  Apply  inflation  factors  to  program  costs  and  funding 

19.  Understand  the  impact  of  inflation  on  program  costs  and  funding 

20.  Understand  the  impact  of  production  rate  and  quantity  decisions  on  program  cost 

21.  Understand  the  impact  of  budget  cuts  on  unit  marginal  cost 

22.  Understand  cost  elements  such  as  direct  labor,  direct  materials,  general  & 
administrative,  profit,  and  overhead 

23.  Understand  ways  to  characterize  costs  such  as  fixedA^ariable  and  recurring/non¬ 
recurring 

24.  Do  break-even  analysis  using  the  concepts  of  fixed  and  variable  costs 

25.  Understand  the  cost  concepts  of  reasonableness,  ailocability,  and  allowability 

26.  Apply  learning  curve  techniques  to  analyze  production  costs 

Please  Turn  to  the  Next  Page 
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Frequency 


1  1 

2 

3 

4 

5 

6 

Don’t  Know 

How  frequently  are  you  called  upon  to: 

27.  Understand  the  impact  of  learning  curves  on  production  costs 

28.  Develop  a  Work  Breakdown  Stmcture  (WBS)  that  describes  the  entire  work  effort 

29.  Understand  the  uses  of  a  WBS  for  cost  management 

30.  Understand  the  role  the  organization  plays  in  the  Planning,  Programming,  and 
Budgeting  System  (PPBS) 

31.  Understand  the  congressional  budgeting  and  appropriations  processes 

32.  Understand  the  flow  of  funds  through  the  expenditure  categories  of  commitments, 
obligations,  and  expenditures 

33.  Distribute  the  cost  of  work  packages  across  the  time  horizon  to  develop  a 
performance  measurement  baseline  (PMB) 

34.  Understand  the  composition  of  a  PMB 

35.  Evaluate  contractor  cost  accounting  and  control  systems 

36.  Understand  contractor  cost  accounting  and  control  systems 

37.  Select  the  appropriate  contract  type  for  a  project 

38.  Understand  the  cost  implications  of  alternative  contract  types  and  pricing  mechanisms 

39.  Understand  the  impact  of  the  political  environment  on  acquisition  management 

40.  Generate  congressionally  required  reports  such  as  the  SAR 

4 1 .  Understand  congressional  reporting  requirements 

42.  Understand  the  legal  and  regulatory  requirements  for  cost  and  schedule  control 
systems  such  as  CVSCSC 

43.  Understand  the  impact  of  changes  in  scope  on  the  cost  of  defense  contracts 

44.  Estimate  earned  value  using  methods  such  as  weighted  milestones  and  percent 
complete 

45.  Understand  the  concept  of  earned  value  and  methods  for  calculating  it 

46.  Develop  an  estimate  at  completion  (EAC)  based  on  the  data  presented  in  contractor 
performance  reports 

47.  Understand  the  use  of  estimates  at  completion  (EAC)  in  cost  management 

48.  Analyze  contractor  reports  such  as  the  CPR  and  CSSR 

49.  Understand  contractor  cost  reports  such  as  the  Cost  Performance  Report  (CPR)  and 
Cost  /  Schedule  Stams  Report  (CSSR) 

50.  Evaluate  contractor-recommended  corrective  actions  and  select  an  appropriate  course 
of  action 

5 1 .  Develop  corrective  actions  to  counter  unfavorable  program  variances 


Please  Turn  to  the  Next  Page 
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Sections;  Importance 

In  this  section  you  will  evaluate  program  manager  competencies  in  terms  of  importance. 
Please  note  the  response  scale  when  answering  the  questions. 


Importance 


1 

2 

3 

4 

5 

6 

Not 

Important 

Slightly 

Important 

Important 

Very 

Important 

Extremely 

Important 

Don’t  Know 

How  important  is  it  for  you  to: 

52.  Be  able  to  apply  time  value  of  money  techniques  such  as  return  on  investment  (ROI), 
net  present  value  (NPV),  internal  rate  of  return  (IRR),  and  Discounted  Cash  Flow  (DCF) 

53.  Understand  the  impact  of  the  time  value  of  money  on  financing  and  budgeting 

54.  Understand  how  contractors  secure  funds  to  support  ongoing  projects,  plant 
improvements,  and  new  product  development 

55.  Be  able  to  develop  a  cost  estimate  using  appropriate  methods  (e.g.  parametric, 
analogy,  grass  roots) 

56.  Understand  cost  estimates  developed  using  appropriate  methods  (e.g.  parametric, 
analogy,  grass  roots) 


U3 


Comments 

•  Do  you  feel  that  any  significant  program  manager  cost  management  competencies 
were  omitted? 


•  What  single  activity  related  to  cost  management  is  the  most  important  to  you  and  what 
activity  is  most  ftequently  used? 


•  Any  other  comments? 
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Importance 


ndixC:  Ranked  Data 


Importance 


Appendix  D:  Decision  Rule  Application  to  Total  Sample 


Median  Sum 

Valuable 

Median 

Valuable 

1.1 

7 

X 

2.1 

4 

12 

4 

2.2 

7 

X 

1.3 

5 

2.3 

8 

X 

1.4 

3 

2.4 

7 

X 

1.5 

5 

2.5 

7 

X 

1.6 

4 

2.6 

4 

1.7 

5 

2.7 

6 

X 

1.8 

7 

X 

2.8 

4 

1.9 

4 

2.9 

6 

X 

1.10 

7 

X 

2.10 

5 

1.11 

3 

2.11 

6 

X 

1.12 

6 

X 

2.12 

7 

X 

1.13 

7 

X 

2.13 

4 

1.14 

5 

2.14 

6 

X 

1.15 

6 

X 

3.1 

6 

X 

1.16 

6 

X 

3.2 

7 

X 

1.17 

6 

X 

3.3 

6 

X 

1.18 

7 

X 

3.4 

4 

1.19 

7 

X 

3.5 

4 

1.20 

4 

16 

7 

X 

1.21 

6 

X 

3.7 

7 

X 

1.22 

4 

3.8 

7 

X 

1.23 

6 

X 

3.9 

8 

X 

3.10 

7 

X 
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Appendix  E:  Don*t  Know  Percentages 

Frequency  Don*t  Know  Percent 


Importance  Don’t  Know  Percent 


II9 


Competency 

Importance 
P- Value 

Frequency 

P-Value 

1.1 

.8137 

.6806 

1.2 

.8777 

.8653 

1.3 

.9334 

.8409 

1.4 

.1817 

.2481 

1.5 

.6922 

.8823 

1.6 

.1684 

.8350 

1.7 

.7550 

.6906 

1.8 

.4016 

.0418 

1.9 

.8712 

.3492 

1.10 

.9106 

.1294 

1.11 

.4163 

.3364 

1.12 

.4103 

.4593 

1.13 

.8538 

.4097 

1.14 

.8721 

.4104 

1.15 

.3394 

.8519 

1.16 

.3545 

.8791 

1.17 

.1280 

.2892 

1.18 

.5879 

.0761 

1.19 

.4105 

.3986 

1.20 

.4979 

.8460 

1.21 

.3166 

.1915 

1.22 

.4237 

.9391 

1.23 

.0192 

.7547 

Management  Education 


Competency 


Importance 

P-Value 

Frequency 

P-Value 

.6143 

.3589 

.8207 

.7369 

.2926 

.1335 

.5357 

.1022 

.8601 

.1536 

.5479 

.5202 

.6040 

.0595 

.7557 

.6569 

.7731 

.9404 

.1477 

.1953 

.3258 

.1077 

.9418 

.0843 

.1234 

.6243 

.3532 

.8556 

.7086 

.3483 

.9214 

.3486 

.2548 

.0938 

.2590 

.8256 

.4218 

.9879 

.6626 

.6214 

.2544 

.4238 

.6225 

.5461 

.9590 

.6222 

.8193 

.1616 

t 
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Organization  Type 


Competency 

Importance 
P- Value 

Frequency 

P-Value 

1 

Competency 

Importance 

P-Value 

Frequency 

P-Value 

1.1 

.3028 

.0189 

2.1 

.1189 

.8170 

1.2 

.2543 

.5881 

2.2 

.9539 

.1940 

1.3 

.6534 

.9117 

2.3 

.8439 

.1911 

1.4 

.9378 

.0477 

2.4 

.4573 

.0645 

1.5 

.9616 

.3205 

2.5 

.5781 

.0604 

1.6 

.3190 

.3210 

n 

2.6 

.1988 

.6920 

1.7 

.2618 

.5466 

2.7 

.3779 

.9526 

1.8 

.3247 

.7802 

2.8 

.9499 

.8609 

1.9 

.2542 

.5054 

2.9 

.6400 

.5375 

1.10 

.8798 

.0968 

2.10 

.2790 

.0053 

1.11 

.1782 

.7945 

2.11 

.3810 

.0525 

1.12 

.6884 

.7502 

2.12 

.1206 

.5525 

1.13 

.7078 

.1538 

2.13 

.5210 

.1180 

1.14 

.4235 

.0688 

2.14 

.5792 

.1697 

1.15 

.5204 

.0463 

3.1 

.5496 

.9482 

1.16 

.5859 

3.2 

.3418 

.5013 

1.17 

.7494 

.0818 

3.3 

.9240 

.3169 

1.18 

.5478 

.0131 

3.4 

.9870 

.4726 

1.19 

.9027 

.0236 

3.5 

.8797 

.4002 

1.20 

.1803 

.3227 

3.6 

.2520 

.0499 

1.21 

.7991 

.9355 

3.7 

.7029 

.0108 

1.22 

.8847 

.0481 

3.8 

.6909 

.0005 

1.23 

.6866 

.1143 

3.9 

.5361 

.0042 

3.10 

.8421 

.1206 
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Primary  Program  Activity 


Competency 

Importance 
P- Value 

Frequency 

P-Value 

1 

Competency 

Importance 
P- Value 

Frequency 

P-Value 

1.1 

.8173 

.3319 

2.1 

.9542 

.1426 

1.2 

.5404 

.5418 

2.2 

.2401 

.1661 

1.3 

.6946 

.8972 

2.3 

.2856 

.3502 

1.4 

.9892 

.1988 

2.4 

.6014 

.4427 

1.5 

.4867 

.3958 

2.5 

.6728 

.1220 

1.6 

.6419 

.5655 

2.6 

.7441 

.5388 

1.7 

.6191 

.9233 

2.7 

.1020 

.8704 

1.8 

.4642 

.8464 

2.8 

.9935 

.6625 

1.9 

.3681 

.9737 

2.9 

.0730 

.7983 

1.10 

.5214 

.2059 

2.10 

.0152 

.0119 

1.11 

.5596 

.2157 

2.11 

.0506 

.2338 

1.12 

.0575 

.9259 

2.12 

.3216 

.8915 

1.13 

.2567 

.2798 

2.13 

.7447 

.2734 

1.14 

.4321 

.5584 

2.14 

.0132 

.5185 

1.15 

.6779 

.7904 

3.1 

.1595 

.4435 

1.16 

.0341 

.0599 

3.2 

.0124 

.7189 

1.17 

.0029 

.5248 

3.3 

.0573 

.1826 

1.18 

.1873 

.2612 

3.4 

.4950 

.2718 

1.19 

.4353 

.0977 

3.5 

.7385 

.0544 

1.20 

.3733 

.6931 

3.6 

.0517 

.1701 

1.21 

.7993 

.1145 

3.7 

.0758 

.0435 

1.22 

.4509 

.6471 

3.8 

.3186 

.0062 

1.23 

.3407 

.6386 

3.9 

.2833 

.0112 

3.10 

.2959 

.0794 

j 


122 


Position  Certification  Level 


Competency 

Importance 
P- Value 

Frequency 

P-Value 

1 

Competency 

Importance 

P-Value 

Frequency 

P-Value 

1.1 

.2788 

.0932 

2.1 

.0728 

.5908 

1.2 

.7593 

2.2 

.4160 

.1768 

1.3 

.4250 

2.3 

.4584 

.3722 

1.4 

.7003 

.3011 

2.4 

.2265 

.1116 

1.5 

.2590 

.1525 

2.5 

.3397 

.1589 

1.6 

.1285 

.3230 

2.6 

.6358 

.2271 

1.7 

.2330 

.7035 

2.7 

.2648 

.0586 

1.8 

.4503 

.0407 

2.8 

.9452 

.6920 

1.9 

.4055 

.2389 

2.9 

.5353 

.8166 

1.10 

.1711 

.4324 

2.10 

.7318 

.4863 

1.11 

.1505 

.5424 

2.11 

.7339 

.3254 

1.12 

.4040 

.0224 

2.12 

.4836 

.7371 

1.13 

.1282 

.5868 

2.13 

.1532 

.5833 

1.14 

.0952 

.0790 

2.14 

.4400 

.2056 

1.15 

.2723 

.2806 

3.1 

.1550 

.1167 

1.16 

.5470 

.6104 

3.2 

.8653 

.3945 

1.17 

.1016 

.7869 

3.3 

.0087 

.1768 

1.18 

.4217 

.6433 

3.4 

.3577 

.1935 

1.19 

.9809 

.8799 

3.5 

.9768 

.2410 

1.20 

.6683 

.1311 

3.6 

.0462 

.0110 

1.21 

.1897 

.3358 

3.7 

.1617 

.0113 

1.22 

.0306 

.1470 

3.8 

.7817 

.0183 

1.23 

.2519 

.3725 

3.9 

.4822 

.0299 

3.10 

.5357 

.0343 
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ndix  G:  ComDetencv  Rankins  for  Groun  Comnarison 


%  4  or  5  with 
Alternate  Data 


46.2 


45.9 


42.3 


37.8 


35.1 


35 


28.1 


27.8 


27.6 


25.4 


24.9 


24.6 


23.7 


22 


19.5 


19.1 


18.3 


17.1 


16.9 


16.5 


16 


14.4 


14 


13.5 


12.7 


12.1 


11.4 


11.1 


10.4 


9.6 


9.3 


2 


8.6 


Competency 

%  4  or  5 

Competency 

%  4  or  5  with 
Alternate  Data 

1.3 

6.9 

1.3 

6.9 

1.7 

5.8 

1.7 

6.6 

2.101 

5.7 

2.101 

5.7 

3.5 

5.7 

3.5 

5.7 

1.2 

4.5 

1.2 

4.5 

3.8 

3.8 

3.8 

3.8 

1.201 

3.4 

1.201 

3.4 

2.6 

3.4 

2.6 

3.4 

2.8 

2.6 

2.8 

2.6 

1.4 

2 

1.22 

2 

1.22 

2 

\ 

2.13 

2 

2.13 

2 

\ 

2.1 

1.9 

2.1 

1.9 

N 

1.4 

1.2 

1.9 

0 

1.9 

0 
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Importance 
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Competency 

%  4  or  5 

Competency 

%  4  or  5  with 
Alternate  Data 

1.7 

42.6 

2.1 

40.4 

2.1 

40.4 

2.8 

34.8 

2.8 

34.8 

1.7 

33 

1.6 

WEEEM 

3.5 

32.4 

3.5 

32.4 

1.14 

32.1 

1.14 

32.1 

1.6 

31.2 

1.201 

28.5 

1.201 

28.5 

2.6 

28.2 

2.6 

28.2 

1.9 

25.9 

1.9 

25.9 

1.22 

25.8 

1.22 

25.8 

3.4 

24.3 

3.4 

24.3 

1.4 

22.2 

1.11 

22 

1.11 

22 

’  ^ 

1.4 

16.5 
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